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ABSTRACT 

During  January  1979  VP-31,  the  West  Coast  P-3  Fleet 
Replacement  Squadron,  implemented  an  Instructional  System 
Development  based  training  program  to  replace  their 
traditional  training  program.  Due  to  monetary,  manpower,  and 
time  constraints,  the  evaluation  phase  of  the  new  training 
program  was  not  completely  developed  or  implemented.  This 
thesis  examines  the  current  status  of  the  external  evaluation 
portion  of  the  new  training  program  in  an  attempt  to 
determine  the  feasibility  of  its  completion  and 
implementation.  The  external  evaluation  plan  is  related  to 
the  Interservice  Procedures  for  Instructional  System 
Development  Model.  From  this  analysis,  a  better 
understanding  of  the  plan  is  gained  and  recommendations  for 
an  improved  external  evaluation  program  and  training  system 
are  presented. 
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I.   INTRODUCTION 

During  the  last  few  decades,  concentrated  efforts  have 
been  directed  toward  improving  the  design  of  Navy  training 
programs.  One  of  the  most  significant  achievements  evolving 
from  exhaustive  research  conducted  in  management  sciences, 
communication  sciences,  and  behavioral  sciences  has  been  the 
application  of  systematic  procedures  to  instructional  system 
design.  The  Navy  began  applying  the  systematic  process  to 
the  design  of  training  programs  in  the  late  1950' s. 
Continued  refinement  of  the  initially  crude  systematic  proce- 
dures eventually  led  to  the  modern  day  state-of-the-art  "sys- 
tems approach"  methodology  for  design  of  training  systems. 

The  systems  approach  as  utilized  in  instructional  design 
is  a  process  of  planning  instruction  which  makes  use  of 
research  and  learning  theory  and  employs  empirical  testing 
data  as  a  means  for  the  improvement  of  the  designed 
instruction  [Ref.  1] .  Systems  approach  technology  was  the 
impetus  for  the  Instructional  Systems  Development  (ISD) 
program  which  is  the  approach  now  required  by  the  Navy,  Air 
Force,  Army,  and  Marines  for  instructional  system  design. 
Instructional  systems  development  is  a  systematic  planning 
and  development  process  which  is  designed  to  optimize 
training  effectiveness  and  efficiency.  Theoretically  it 
seems   to  be  a  far  better  method  to  design  training   programs 


than  methods  previously  employed.  This  is  especially  true  in 
the  area  of  quality  control,  because  unlike  traditional 
design  methods,  the  ISD  approach  places  tools  within  a 
training  system  to  identify  and  evaluate  its  design  errors. 
This  procedure  is  especially  important  when  the  system  is 
large  and  the  training  complex. 

Since  January  of  1979,  Patrol  Squadron  Thirty-one  (VP- 
31),  which  is  the  Navy's  West  Coast  P-3  Orion  Fleet 
Replacement  Squadron  (FRS) ,  has  been  utilizing  the  ISD 
process  to  train  flight  crews  for  the  P-3  aircraft.  This 
process  was  also  adopted  by  Patrol  Squadron  Thirty  (VP-30) , 
the  East  Coast  FRS  in  January  of  1980.  Development  of  the  P-3 
Instructional  System,  the  VP  FRS  version  of  the  ISD  was 
accomplished  over  several  years  by  a  large  number  of  Navy  and 
contractor  personnel.  The  system  consists  of  a  series  of 
orderly,  logical,  and  interrelated  steps  to  produce  a 
training  course  which  is  efficient  and  effective  in  providing 
graduates  with  the  skills,  knowledge,  and  attitudes  essential 
to  the  performance  of  a  job  [Ref.  2], 

Since  the  introduction  of  the  P-3  Instructional  System  in 
1979,  the  quality  control  phase  has  never  been  formally 
implemented  due  to  monetary  and  manpower  constraints.  These 
undesirable  limitations  are  common  to  many  situations  in 
times  of  economic  uncertainty.  The  lack  of  an  active 
evaluation  phase  changes  the  structure  of  the  system  from   an 
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ISD  process  to  an  extremely  sophisticated  and  rigidly 
structured  training  program  with  no  means  available  for 
rigorous  evaluation  of  the  system.  Therefore,  even  though 
the  system  may  be  well  designed  and  expertly  managed,  there 
is  no  feasible  way  to  determine  whether  or  not  the 
instructional  system  objectives  and  required  fleet 
performance  levels  are  being  met.  The  purpose  of  this  thesis 
is  to  examine  the  status  of  the  designed  quality  control 
program  and  provide  recommendations  for  its  completion  and 
incorporation.  As  a  result  of  this  examination,  an  improved 
P-3  FRS  training  system  should  be  realized. 

A.   BACKGROUND 

Prior  to  the  implementation  of  ISD  training  procedures, 
VP  fleet  replacement  squadrons  utilized  traditional  training 
methods  and  media,  consisting  primarily  of  the  group-paced 
lecture/demonstration  concepts.  These  concepts,  unlike  ISD, 
offered  no  affordable  means,  of  locating  training  design 
errors,  to  ensure  the  existance  of  an  optimal  instructional 
system.  This  section  will  discuss  the  traditional  VP  FRS 
training  program  pointing  out  some  of  the  serious 
shortcomings  that  led  to  the  need  for  a  more  sophisticated 
systems  oriented  training  process.  A  brief  overview  of  ISD 
will  then  be  presented,   to  demonstrate  how  most   traditional 
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training   system  problems  can  be  solved  by  using  the  closed- 
loop  "systems  approach"  for  training  system  design. 

1.   The  VP  Fleet   Replacement   Squadron   Traditional 
Training  Program 

The  traditional  training  system  used  by  the  VP  Fleet 
Replacement  Squadrons  until  the  late  1970' s,  was  well 
designed  and  skillfully  administered.  However,  inherent  in 
the  design  methods  of  the  program  was  the  problem  of 
measuring  training  effectiveness.  Training  managers  were  at 
best  able  to  make  only  broad  assessments  of  the  activities' 
training  value,  and  their  methods  were  not  systematic, 
comprehensive,  objective,  nor  scientifically  accurate. 
Therefore  this  type  of  evaluation  proved  to  be  of  no 
particular  value  to  either  training  personnel  or  command 
level  managers  [Ref.  3]. 

Evaluation  of  a  training  program  is  a  critical 
concept  needed  to  determine  if  the  objectives  of  the  program 
and  needs  of  the  organization  are  being  met.  Also,  it  must 
determine  if  the  program  is  in  fact  providing  the  necessary 
information  for  graduates  to  do  their  job.  Evaluation 
overrides  the  human  error  inevitable  in  the  application  of 
subjective  opinion  in  providing  a  means  for  training  activity 
personnel  to  apraise  themselves,  their  practices  and 
products.    The   evaluation  must  be   specific,   continuous, 
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cooperative,  conducted  in  terras  of  purposes,  and  based  on 
objective  methods  and  standards  [Ref.  3], 

Recognizing  the  need  to  determine  his  training 
activities'  effectiveness  and  efficiency,  the  commanding 
officer  of  Patrol  Squadron  Thirty-one  requested  assistance 
from  the  Naval  Training  Device  Center  (NTDC)  in  September 
1971.  The  Naval  Training  Device  Center  designed  an 
evaluation  program  encompassing  the  total  training  function 
of  the  FRS.  The  program  was  modeled  after  similiar  studies 
of  other  naval  aviation  training  programs.  The  results  of 
the  evaluation,  known  as  the  HVP-31  Educational  Self -Audit" 
sought  to  determine  the  quality  of  specific  aspects  of  the 
VP-31  training  program  and  to  encourage  the  training  staff 
and  faculties  to  develop  improved  instructional  systems  and 
materials. 

The  evaluation  program,  based  upon  systematic  and 
detailed  procedure,  was  divided  into  six  related  segments; 
(1)  Philosophy  and  Goals,  (2)  Buildings  and  Facilities,  (3) 
Management,  (4)  Staff  and  Faculty,  (5)  Curriculum,  and  (6) 
Instructional  Support.  The  extensive  analysis  of  each 
segment  identified  several  basic  training  discrepancies 
within  the  FRS  program.  The  evaluation  team  concluded  that 
the  most  important  problems  would  be  corrected  and  training 
at  VP-31  would  be  significantly  improved  if  the  following 
changes  were  made:    (1)   Provide   professional   educational 
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assistance  on  a  full-time  basis,   (2)  provide  more  complete 

dialogue   among   all  levels  of  the  Air  Antisubmarine  Warfare 

(AASW)  community,  (3)  improve  overall  instructional  skills  of 

instructors   and   instructor   supervisors,   (4)   implement   a 

program   leading   to   increased   use  of   self-instructional 

materials   and  media,   (5)  establish  revision  procedures   to 

stabilize  the  curriculum,  (6)  decrease  the  rate  of  instructor 

turn-over,  and  (7)  establish  better  communication  between  VP- 

31  and  all  segments  of  the  AASW  community. 

The  above  problems,   found  by  the  NTDC  team,   coupled 

with  numerous  other  shortcomings  such  as  no  clearly  written 

statements   of   philosophy  and  goals,    no   statement   of 

management  objectives,   etc.,   led  to  the  general  conclusion 

that  a  better  training  system  was  needed  at  VP-31.   This 

decision  was   strongly  reinforced  in  1974  when   it   became 

apparent   that   the   traditional  training  program  could   not 

possibly  handle   the   upcoming  incorporation  of   the   2F87F 

flight  simulator  or  withstand  a  proposed  twenty-three  percent 

reduction  in  aircraft  and  manpower.    The  Commanding  Officer, 

Patrol  Squadron  Thirty-one  [Ref.   4]  in  submitting  a  proposal 

for  instructional  development  of  P-3  FRS  training  stated: 

"...  This  effort  will  require  the  complete  restructuring 
of  the  present  training  program  both  in  concept  and  format. 
A  revision  of  such  magnitude  requires  a  formal  development 
program.  Unfortunately,  major  development  efforts  in  the 
past  employed  non-scientific  methods  and  resulted  in  a  R/S 
(Readiness  Squadron)  training  program  of  questionable 
efficiency  which  left  the  R/S  extremely  vulnerable  to 
reductions   during   periods  of  fiscal/personnel   austerity. 

14 


If  the  new  R/S  training  program  is  to  teach  the  knowledge 
and  skills  essential  for  successful  fleet  performance  and 
be  supportable  on  the  basis  of  cost-effectiveness,  the 
process  by  which  it  is  developed  must  include  the 
deliberate,  orderly  methods  of  a  systems  approach  to 
training." 

This  was  the  first  step  toward  application  of  ISD 
methods  for  training  system  development  within  the  VP 
community. 

2.   Instructional  Systems  Development 

The  Random  House  Dictionary  defines  instruction  as 
the  art  or  practice  of  furnishing  knowledge,  especially  by  a 
systematic  method.  More  specifically,  it  is  a  set  of  events 
employed  by  those  whose  purpose  it  is  to  develop  an  organized 
plan  to  facilitate  learning.  Instruction  then,  simply 
stated,  is  a  means  to  help  people  learn. 

Instruction  is  often  applied  to  an  area  as  an 
organized  way  of  accomplishing  certain  goals.  This  area  of 
interest  in  the  most  general  terms  has  no  fixed  boundaries 
and  is  often  referred  to  as  a  system.  System  is  a  relative 
term  and  can  be  thought  of  as  composites  made  up  of 
interacting  constituent  parts  [Ref.  5],  From  these  two  ideas 
emerges  the  term  instructional  system,  whose  purpose  can  be 
thought  of  as  furnishing  a  means  to  satisfy  specified 
objectives  to  reach  any  outcome  required  by  a  learning 
program.  According  to  AFM-50  [Ref.  6],  the  Air  Force  manual 
for  instructional  system  development,  an  instructional  system 
can   be  thought  of  as  an  integrated  combination  of   resources 
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(students,  instructors,  materials,  equipment,  and 
facilities) ,  techniques,  and  procedures  performing 
efficiently  the  functions  required  to  achieve  specified 
learning  objectives.  For  the  purpose  of  this  discussion,  an 
instructional  system  will  be  considered  a  device  for 
attaining  whatever  learned  outcomes  a  training  program 
intends . 

Instructional  design  can  be  a  very  complex  and  time- 
consuming  process.  The  fact  that  there  are  numerous  ways  to 
approach  this  seemingly  impossible  task  only  serves  to 
complicate  matters.  Gagne ■  and  Briggs  [Ref.  1],  two  pioneers 
in  learning  concepts  and  instructional  design,  present  one 
method  of  design  that  they  consider  feasible  and  worthwhile. 
The  process  they  describe  is  based  upon  the  following  five 
important  assumptions.  First,  the  design  must  be  aimed  at 
aiding  the  learning  of  the  individual.  Although  learners  are 
often  assembled  into  groups,  learning  occurs  within  each 
member  of  a  group.  Second,  instructional  design  contains 
both  immediate  and  long-range  phases.  The  instructor 
preparing  lesson  plans  several  hours  before  a  presentation 
constitutes  the  immediate  phase,  while  the  long-range 
considerations  may  consist  of  a  set  of  lessons  covering  a 
topic  and  a  set  of  topics  covering  a  course.  The  third 
assumption  is  the  belief  that  systematically  designed 
instruction  can  greatly  affect  individual  human  development. 
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This  "guiding  light"  concept  ensures  that  all  students  are 
given  an  equal  opportunity  to  use  their  individual  talents  to 
the  fullest  degree.  The  fourth  idea  specifically  states  that 
the  design  of  instruction  should  be  conducted  by  means  of  a 
systems  approach.  This  idea  will  be  discussed  further  in  the 
latter  part  of  this  section.  The  fifth  and  one  of  the  most 
important  points  is  that  when  designing  instruction,  one  must 
take  into  account  how  human  beings  learn  in  every  phase  of 
design. 

The  aspects  of  learning  theory  which  have  a  direct 
impact  on  instruction  are  those  which  relate  to  controllable 
events  and  conditions  [Ref.  1] .  Continued  research  in  the 
area  of  human  learning  has  brought  to  light  more 
sophisticated  theories  than  were  previously  held.  It  is  now 
known  that  two  categories  of  factors  exist  in  a  learning 
situation.  First  are  the  age-old  factors  contiguity, 
repetition,  and  reinforcement  that  exist  in  an  individual's 
external  environment.  The  second  and  newest  category  of 
factors  are  internal  in  nature.  That  is,  they  originate  in 
an  individual's  memory.  Some  of  the  most  important  internal 
factors  are:  factual  information,  intellectual  skills, 
strategies,  motivation,  and  an  attitude  of  confidence.  These 
new  state-of-the-art  concepts  depicted  in  Figure  1.1 
emphasizes  the  fact  that,  in  the  design  of  instruction,  both 
the   external   situation   stimulating  the  learner   and   those 
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capabilities   which  learners  bring  with  them   must   be 
considered  [Ref.  1] . 
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Factors  Affecting  the  Learning  Event 


As  stated  earlier  in  the  Gagne*  and  Briggs1  design 
model  assumptions t  instructional  design  should  be  constructed 
by  means  of  a  systems  approach.  This  revolutionary  concept 
has  proven  to  be  an  invaluable  tool  capable  of  incorporating 
the  high  technology  innovations  developed  during  the  last  two 
decades  into  a  cohesive  plan  for  the  improvement  of  training 
effectiveness  and  efficiency.  Rapid  development  in  the  area 
of  educational  technology  produced  such  items  as  programmed 
instruction,  the  proficiency  test,  and  computer  assisted 
instruction  in  such  a  short  timeframe,  that  attempts  to 
effectively  and  economically  incorporate  all  of  these  ideas 
into  any  sort  of  an  instructional  program  was  impossible. 
With   the   timely  development  of   the   systems  approach,  it 
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became  possible  to  incorporate  these  innovative  possibilities 
into  workable  educational  plans  [Ref.  5]. 

While  systematic  problem  solving  is  fairly  old,  it 
was  not  until  after  the  Second  World  War  that  such  concepts 
began  being  used  in  the  field  of  education.  Following  the 
War,  individuals  who  had  successfully  applied  the  systems 
approach  to  the  design  and  implementation  of  complex  military 
equipment  returned  to  the  field  of  education  and  began  using 
this  new  technique  to  design  instructional  systems. 

The  systems  approach  is  a  method  for  constructing  a 

plan  that  has   the  capability  to  design,   implement,   and 

evaluate  a  program  that  has  been  constructed  to  achieve 

certain  goals.  Gagne'  and  Briggs  [Ref.  1]  give  the  following 

description  of   the  systems  approach  as  utilized  in  their 

instructional  design  plan. 

"...The  systems  approach  to  instructional  design  involves 
the  carrying  out  of  a  number  of  steps  beginning  with  an 
analysis  of  needs  and  goals,  and  ending  with  an  evaluated 
system  of  instruction  which  demonstrably  succeeds  in 
meeting  accepted  goals.  Decisions  in  each  of  the 
individual  steps  are  based  upon  empirical  evidence,  to  the 
extent  that  such  evidence  allows.  Each  step  leads  to 
decisions  which  become  'inputs'  to  the  next  step,  so  that 
the  whole  process  is  as  solidly  based  as  is  possible  within 
the  limits  of  human  reason.  Furthermore  each  step  is 
checked  against  evidence  that  is  'fed  back'  from  subsequent 
steps  to  provide  an  indication  of  the  validity  of  the 
system." 

In  practice,  the  systems  approach  provides  an  orderly  and 

logical  way  of  reaching  pre-set  goals.   It  is  not  a  difficult 

concept   in  theory,   but  actually  applying  the  approach  to 
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improve  the  operation  and  success  of  a  system  can  prove  to  be 
both  time  consuming  and  complex.  Effective  utilization  of 
the  concept  requires  an  individual  with  a  solid  understanding 
of  the  method  and  an  enormous  amount  of  past  experience  in 
the  design  of  each  phase.  Two  reasons  for  the  limited 
success  of  the  systems  approach  in  the  area  of  training  in 
the  past  are:  first,  the  instructional  field  is  vastly  more 
complex  than  any  area  where  the  concept  has  been  used  in  the 
past,  and  second,  few  training  system  designers  have  a 
reasonable  grasp  of  the  approach.  The  complexity  of  the 
system  becomes  readily  apparent  if  one  considers  all  of  the 
alternate  solutions  available  at  each  step  of  the  process  and 
identification  of  the  possible  effects  of  each  one.  This 
complexity  should  not  discourage  usage  of  the  system.  Once 
the  concepts  are  mastered  and  enough  practice  has  been 
obtained  in  this  method  of  instructional  design,  it  is  by  far 
the  most  effective  tool  we  have  available  today  to  design 
programs  to  enhance  human  learning. 

Klaus  [Ref.  5]  states  that  most  designers  of 
instructional  systems  agree  on  the  following  six  steps  as  the 
principle  characteristics  of  a  systems  approach  to 
instructional  design:  (1)  recognition  and  definition  of  the 
problem  in  terms  of  the  desired  outcome  of  any  system  action, 
(2)  establishment  of  the  criteria  to  be  used  in  assessing  the 
success   of   a  change,    (3)   identification  of   potential 
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solutions  to  the  problem,  (4)  feasibility  testing  for  each 
promising  solution,  (5)  begin  assembling  the  new  system 
adopting  one  or  more  of  the  solutions,  and  (6)  provide 
procedures  for  improving  the  system  as  experience  is  gained 
and  limitations  or  short  falls  are  discovered.  This  closed- 
loop  systems  approach  is  illustrated  in  Figure  1.2.  Note  the 
closed-loop  and  continuous  cycle  of  the  system.  The  dashed 
arrow  represents  the  ability  of  the  system  to  insert  a 
discovered  problem  into  the  loop  to  institute  an  improvement 
cycle.  The  system  continually  corrects  itself  everytime  a 
problem  is  discovered.  Thus,  in  theory,  the  longer  a  system 
is  in  operation  the  better  it  will  become.  Although  there 
are  several  slightly  varying  concepts  of  what  constitutes  a 
systems  approach,  this  is  one  of  the  most  highly  reguarded. 
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Source:   Ref.5:  p.  179 
Figure  1.2.   Steps  in  the  Systems  Approach  Cycle 


The  Armed  Services  began  full-scale   utilization  of 
the   systems  approach  to  training  around  1970,   when  the  Air 
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Force  adopted  a  training  development  concept  called 
Instructional  Systems  Development  (ISD) .  The  ISD  process  is 
a  broad  application  of  the  systems  approach  to  training  which 
by  design  increases  the  quality  of  training  and  reduces  the 
amount  of  time  and  money  necessary  to  train  an  individual.  A 
common  misconception  that  should  be  corrected  from  the 
beginning  is  the  fact  that  ISD  is  a  process  through  which 
training  alternatives  are  selected,  and  should  not  be 
confused  with  or  put  on  the  same  level  as  tools  of 
contemporary  training  technology,  such  as  slide-tape 
presentations,  programmed  instruction,  computer  assisted 
instruction,  etc.  Instructional  Systems  Development,  in  a 
broad  sense,  is  a  process  which  allows  for  the  orderly 
development  or  change  of  training  programs  and  ensures  cost- 
effective  instruction  that  produces  graduates  capable  of 
acceptable  performance  on  the  job  [Ref.  6].  Since  it  is  an 
application  of  the  systems  approach,  practical  use  of  the 
process  involves  the  cyclic  concept  suggested  in  Figure  1.2. 
As  illustrated,  in  the  figure,  the  constant  revision  of  past 
steps  along  with  projections  of  future  effects  serve  to 
produce  a  much  stronger  and  improved  system.  Empirical 
testing  of  the  system  solidifies  the  process  and  provides 
maximum  effectiveness.  Through  this  process  designers  can 
actually  set  design  objectives  describing  acceptable  evidence 
of   successful   system  operation,   and  continue   the  design 
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process  until  that  point  of  quality  has  been  reached 
[Ref.  1].  Instructional  systems  Development  can  then  be 
thought  of  as  a  repetitive  process  of  analysis,  design, 
verification,  and  revision.  While  reviewing  the  methodology 
and  application  of  ISD  in  the  armed  services,  Vineberg  and 
Joyner  [Ref.  7]  characterize  the  sequence  of  procedures 
constituting  ISD  as  follows:  (1)  the  rigorous  derivation  of 
training  requirements  from  job  requirements,  (2)  the 
selection  of  instructional  strategies  so  that  the  efficiency 
of  training  is  maximized,  and  (3)  interative  trial  and 
revision  of  instruction  during  development  until  training 
objectives  are  met. 

Utilizing  the  ISD  process  for  instituting  or  revising 
training  within  the  military  services  is  strongly  supported 
by  the  Department  of  Defense.  Within  the  military 
establishment  there  exists  several  models  or  variations  of 
the  model  constructed  for  standardization  of  the  ISD  process 
for  the  armed  forces.  The  standardization  model  is  presented 
in  five  documents  titled  Interservice  Procedures  for 
Instructional  System  Development  (IPISD) ,  and  is  published  by 
the  Navy  as  NAVEDTRA  106A  [Ref.  8].  The  model  was  developed 
for  the  Army  and  later  gained  approval  from  the  Interservice 
committee  on  Instructional  Systems  Development.  It  provides 
detailed  procedures  for  conducting  ISD.  The  Army  uses  this 
exact  model,   distributed  as  TRADOC  Pamphlet  350-30  [REf.   9] 
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for  their  ISD  purposes,  while  the  Navy  has  published  a 
condensed  version  of  the  model  for  its  own  use,  titled 
Procedures  for  Instructional  Systems  Development  in  NAVEDTRA 
110A  [Ref .  10] . 

The  Marine  Corps  uses  a  greatly  reduced  version  of 
the  IPISD  model  that  has  been  altered  to  meet  Marine  Corps' 
documentation  requirements.  Guidelines  can  be  found  in 
Marine  Corps  Order  P151.23B  [Ref.  11].  The  Air  Force  model 
varies  somewhat  from  the  IPISD  model.  Their  version  of  the 
ISD  process  can  be  found  in  Air  Force  Manual  (AFM)  50-2, 
Instructional  System  Development  [Ref.  6]  and  Air  Force 
Pamphlet  (AFP)  50-58,  Handbook  for  Designers  of  Instructional 
Systems  [Ref.  12],  AFM  50-2  provides  an  overview  of  the  Air 
Force's  ISD  model,  while  AFP  50-58  gives  detailed  procedures 
for  implementing  the  concepts  in  AFM  50-2. 

Since  this  thesis  deals  with  Navy  training,  the  focus 
of  this  discussion  will  be  directed  towards  the  IPISD  model. 
As  stated  in  the  letter  of  Promulgation  for  NAVEDTRA  110A 
[Ref.  10]  (the  Navy's  condensed  version  of  IPISD),  the  manual 
" . . .  provides  guidance  for  the  analysis,  design,  development, 
implementation,  and  control  of  instructional  programs  under 
the  direct  cognizance  of  the  Chief  of  Naval  Education  and 
Training  (CNET)."  In  addition  the  letter  states  that,  "All 
curricula  developed  within  or  for  use  within  the  Naval 
Education  and  Training  Command,   except  submarine   training 
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materials  specified  in  CNETINST  1550.14,  will  be  developed  in 
accordance  with  the  policy,  procedures,  and  standards 
contained  therein." 

The  IPISD  model  contains  procedures  that  are 
primarily  concerned  with  the  "how  to  do  it"  aspects  of  ISD. 
The  model  is  based  upon  nine  important  assumptions  that  must 
be  fully  understood  before  attempting  to  use  the  model  for 
instructional  program  design,  revision,  or  administration  of 
an  ISD  training  program.  The  basic  construction  of  the  model 
consists  of  five  phases,  each  being  a  separate  and  distinct 
function.  The  model  is  designed  to  provide  a  sequential 
relationship  of  the  phases. 

Figure  1.3  depicts  a  block  diagram  of  the  IPISD 
model.  Each  phase  is  shown  with  appropriate  steps  to  be 
accomplished.  The  five  phases  of  the  model,  as  shown  in  the 
figure,  are  labeled  Analyze,  Design,  Develop,  Implement,  and 
Control.  Phase  One,  Analyze,  is  the  single  most  important 
step  in  the  ISD  process,  because  it  establishes  what  the 
program  will  teach.  This  phase  provides  the  procedures  to 
define  what  the  jobs  consist  of,  breaking  these  down  into  a 
detailed  listing  of  duties,  tasks,  and  elements  necessary  to 
perform  a  specific  job.  The  analysis  involves  techniques 
such  as  job  interviews,  questionnaires,  experienced 
professionals  using  their  best  judgment,  and  group 
interviews. 
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Figure  1.3.   Five  Phases  of  the  IPISD  Model 
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Phase  Two,  Design,  uses  the  job  analysis  from  Phase 
One  to  begin  designing  instruction.  The  form  and 
specifications  for  training  are  created  and  the 
identification  and  design  of  the  terminal  and  enabling 
objectives  that  an  individual  must  achieve  in  order  to 
satisfactorily  complete  the  program  are  produced.  Phase 
Three,  Develop,  deals  with  the  actual  preparation  of 
instruction.  This  phase  specifies  the  knowledge  and 
performance  learning  activities  that  will  be  used,  develops 
curriculum  outlines  and  instructional  management  plans,  and 
selects  instructional  materials.  When  these  products  have 
been  validated  by  using  empirical  data,  Phase  Four  can  take 
place. 

Phase  Four,  Implementation,  produces  the  steps 
necessary  to  implement  the  instructional  plan  that  was 
developed  in  Phase  Three,  while  Phase  Five,  Control,  provides 
the  procedures  for  the  evaluation  of  the  program  to  determine 
how  effectively  it  has  met  the  initialy  set  goals.  An 
internal  evaluation  is  conducted  as  a  continuous  process  to 
assess  student  performance  and  to  determine  the  effectiveness 
of  the  program  methods  and  materials.  An  external  evaluation 
determines  if  the  course  is  training  graduates  well  enough  to 
perform,   on  the  job,  the  tasks  they  were  trained  to  perform, 
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and  if  these  tasks  are  the  ones  required  to  properly  do  the 
job.  These  evaluations  provide  the  basis  for  program 
revision. 

As  stated  in  NAVEDTRA  106A  [Ref.  8],  the  design  and 
development  of  instruction  must  follow  an  adequate  needs 
analysis.  In  other  words,  when  there  is  a  known  measurable 
discrepancy  between  a  situation  as  it  exists  and  the 
situation  as  it  ought  to  be,  instructional  programs  should  be 
designed  to  eliminate  that  discrepancy.  During  initial 
applications  of  the  ISD  process,  one  should  not  set  out  to 
run  a  course  through  the  process,  but  rather  utilize  the 
process  in  such  manner  that  the  need  for  a  course  becomes 
apparent  through  the  use  of  the  process. 

The  ISD  process  is  an  extremely  sophisticated  method 
of  instructional  development  and  contains  several  unique 
characteristics.  The  process  uses  painstaking  measures  to 
ensure  proper  selection  of  what  is  to  be  trained.  This  is 
accomplished  through  job  surveys  and  experts  from  the  field 
when  a  job  already  exists  and  from  engineering  data  for  new 
jobs  and  systems.  Instructional  Systems  Development  also 
generates  and  applies  alternative  training  methods  selected 
from  state-of-the-art  technology  instead  of  relying  on  tradi- 
tional methodology.  These  advanced  techniques  aid  in  the 
assurance  of  optimal  training  effectiveness,  time  efficiency, 
and  cost.   Thus  there  is  no  need  for  course  content  reduction 
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to  optimize  a  system.  Finally,  one  of  the  most  important 
characteristics  of  the  ISD  process  is  the  provision  for 
measuring  the  quality  of  instruction  and  the  overall  program. 
The  process  uses  test  data  based  on  absolute  standards  of 
performance  to  grade  students  and  check  program  quality.  One 
last  point  that  should  be  noted  about  ISD  methodology  is  its 
flexibility.  When  utilizng  the  process  the  model  can  be 
entered  at  any  step,  or  if  appropriate,  actions  can  be  taken 
in  several  steps  simultaneously. 

It  should  be  clearly  evident  that  the  ISD  process  can 
be  extremely  beneficial  in  the  area  of  instructional  design. 
The  process  can  be  adapted  to  any  training  program  including 
on-the-job  training.  One  can  use  the  process  to  design  new 
instructional  programs  or  to  change  and  improve  existing 
ones.  Since  the  process  has  been  in  use  for  several  years, 
empirical  evidence  now  supports  the  claims  of  improved 
effectiveness,  time  efficiency,  and  costs  with  proper 
application  of  ISD.  Although  many  still  consider  the  process 
an  art,  it  is  by  far  closer  to  a  science  of  education  than 
the  other  approaches  to  instructional  design  that  do  not 
include  measurement  of  their  effects. 

B.   THE  P-3  INSTRUCTIONAL  SYSTEM 

During  the  latter  part  of  September  1974  the  Commanding 
Officer  of  VP-31,  in  a  letter  addressed  to  Commander  Patrol 
Wings,   U.S.   Pacific  Fleet  [Ref.  4],  stated  that  the  P-3  ASW 
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community  was  in  serious  need  of  a  new  training  program  to 
overcome  manpower  and  monetary  shortages  and  still  provide 
quality  training  to  personnel  enroute  to  fleet  squadrons.  He 
went  on  to  say  that  the  training  program  should  be  developed 
by  the  methods  of  a  systems  approach  to  training.  As  justi- 
fication for  requesting  this  particular  type  of  instructional 
system  design,  he  cited  the  Naval  Training  Equipment  Center's 
(NTECs)  Training  Analysis  and  Evaluation  Group  (TAEG)  report 
number  10  [Ref.  13]  which  contains  the  results  of  an  experi- 
mental P-3  pilot  and  flight  engineer  training  program.  The 
experiment  demonstrated  the  efficiency  and  economy  of  a 
systems  approach  to  training  by  indicating  a  potential  media 
and  manpower  savings  of  six  million  dollars  per  year  in  just 
that  one  portion  of  FRS  training.  The  Commanding  Officer  of 
VP-31  further  stated  that  as  trainers  advance  in  sophistica- 
tion, complexity,  and  cost,  the  systems  approach  to  training 
could  be  used  as  a  tool  to  ensure  that  FRS  training  needs  are 
met  and  at  the  same  time  guard  against  the  waste  of  allocated 
funds. 

As  a  solution  to  the  FRS  training  problem,  the  framework 
for  an  instructional  development  program  employing  the 
systems  approach  was  submitted.  The  program  was  patterned 
after  the  S-3  Fleet  Instruction  Team  program,  and  was 
designed  to:  (1)  teach  only  the  essentials  for  successful 
job  performance  in  the  fleet,   (2)  employ  the  latest  and  best 
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technology  in  design  considerations,  and  (3)  maximize  the  use 
of  available  resources  (i.e.,  training  aids,  aircraft,  and 
manpower) . 

With  processing  through  normal  channels  complete  and 
approval  obtained,  Courseware,  Inc.  was  commissioned  along 
with  Navy  and  government  personnel  to  provide  the  expertise 
for  the  new  training  program.  These  developmental  personnel 
were  formed  into  an  Instructional  Systems  Development  Team 
consisting  of  subject  matter  experts  (SMEs) ,  instruction 
psychologists  (IPs) ,  and  technical  assistants  (TAs) .  The 
results  of  their  efforts  provided  training  programs  for  thir- 
teen positions  on  the  P-3  A/B  and  P-3C  aircraft  incorporated 
into  eleven  volumes. 

The  team  of  experts  employed  the  ISD  process  for  the 
program  design,  which  was,  by  this  time,  the  method  required 
by  all  uniformed  services.  Courseware,  Inc.  concurred  with 
this  state-of-the-art  approach,  stating  that  it  offered  the 
best  opportunity  for  effective,  efficient,  and  motivating 
instruction.  Thus  the  philosophy  underlying  the  design  con- 
cept adopted  by  Courseware,  Inc.  was:  determine  what  to 
teach,  determine  how  to  teach  it,  and  determine  whether  or 
not  the  teaching  has  been  successful. 

The  basic  systems  approach  developed  and  validated  by 
Courseware,  Inc.  is  based  upcn  six  assumptions  that  describe 
their  philosophy  on  the  when,   how,   and  why  of  instructional 
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design.  The  ISD  approach  they  developed  for  the  P-3  Instruc- 
tion system  consists  of  ten  major  steps  which  served  as  the 
framework  for  the  project.  The  steps  for  the  most  part  follow 
those  of  the  IPISD  model  presented  in  the  last  section, 
therefore  they  will  not  be   covered  in  any  great  detail. 

Step  one,  job  analysis,  breaks  the  job  down  into  compo- 
nent tasks.  The  level  of  detail  is  kept  relatively  general 
through  this  step.  Step  two,  selection  of  tasks  for 
training,  identifies  those  tasks  listed  in  step  one  that  must 
be  trained  in  the  program.  Step  three,  hierarchical 
analysis,  converts  job  requirements  to  a  set  of  specific 
training  objectives.  This  step  reduces  the  tasks  of  step  one 
to  a  more  specific  level  of  detail.  These  new  tasks  are 
presented  in  the  form  of  a  task  diagram  which  is  used  to  aid 
SMEs. 

Step  four,  media  selection,  provides  media  alternatives 
to  fit  the  learning  requirements  for  each  objective.  The 
media  alternatives  selected  for  the  program  were  workbooks, 
videotape,  slide/tape,  and  simulators.  Step  five,  sequencing 
of  objectives,  constructs  the  order  in  which  training 
materials  will  be  presented  to  the  student.  Step  six,  lesson 
specification,  expands  each  sequenced  objective  to  aid  the 
course  writers.  Step  seven,  authoring  lessons,  creates  sound 
instruction  given  a  specification  of  strategy,   content,   and 
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media.  Step  eight,  production  of  tests,  creates  lesson  tests 
to  measure  student  learning  and  course  effectiveness. 

Step  nine,  training,  is  the  actual  use  of  instructional 
materials  with  students.  Step  ten,  evaluation,  uses  empiri- 
cal means  to  determine  whether  students  are  learning  the 
course  material,  if  students  can  perform  adequately  on  the 
job,  and  if  the  course  is  presenting  the  proper  information 
for  the  job  to  be  completed.  This  step  was  scheduled  to  occur 
during  the  first  year  of  training.  These  ten  steps  provided 
the  basis  for  the  training  system  development. 

Coincident  with  the  completion  of  the  work  established  by 
the  ten  basic  steps  of  the  design  system  was  the  production 
of  the  P-3  Instructional  Systems  Management  Plan.  The  purpose 
of  the  plan  was  to  provide  explicit  guidelines  and  procedures 
to  aid  in  the  implementation  of  the  training  materials 
evolving  out  of  the  development  of  the  new  training  program. 
The  plan  was  also  devised  to  maintain  the  materials  in  an  up- 
to-date  status  as  new  systems  and  procedures  arise  and  as 
revisions  are  required. 

The  management  plan  covers  such  areas  as  administrative 
details,  scheduling,  course  materials,  equipment,  facilities, 
and  quality  control.  Additionally,  the  plan  assigns  specific 
organizational  responsibilities  to  VP-31  and  to  the  Fleet 
Aviation  Specialized  Operational  (FASO)  Training  Group 
Pacific  Fleet  Detachment  Moffett. 
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II.   STATEMENT  OF  PROBLEM 

A.   PROBLEM  STATEMENT 

The  primary  mission  of  the  peace-time  Navy  is  to  train 
for  its  role  as  protector  of  the  seas,  in  time  of  aggression 
against  the  United  States  (U.S.)  or  in  support  of  U.S. 
interests.  The  success  of  the  Navy's  overall  training  system 
is  determined  by  how  well  the  numerous  training  subsystems 
meet  their  pre-established  goals.  The  attainment  of  these 
goals  plays  a  critical  part  in  the  bid  to  reach  and  maintain 
a  high-level  of  Fleet  readiness.  Thus,  designers  of  Naval 
instructional  systems  must  constantly  strive  to  create 
programs  that  are  optimally  effective,  time-efficient,  and  as 
cost-effective  as  possible  to  operate.  The  only  way  to 
insure  the  development  of  such  high-caliber  programs  is  to 
include,  within  the  program,  a  check-and-balance  phase  that 
will  provide  continuous  revision  both  during  design  and  after 
the  program  becomes  operational. 

Since  the  early  1970' s  the  Chief  of  Naval  Education  and 
Training  has  considered  the  measurement  of  training 
effectiveness  through  evaluation  of  trainee  performance,  a 
matter  of  major  concern.  Despite  this  high  level  of  concern, 
the  ability  to  design  and  successfully  incorporate  meaningful 
training  system  appraisal  continues  to  be  a  problem 
throughout   the  Navy  and  all  other  branches  of  the  military 
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service.  The  most  common  denominator  in  this  problem  is,  as 
most  would  expect,  a  lack  of  adequate  assets  to  properly 
operate  the  evaluation  phase  of  instructional  systems,  even 
after  the  phase  has  been  designed  as  an  integral  part  of  the 
training  program. 

The  P-3  Instructional  System  based  on  ISD  methodology, 
was  implemented  as  the  FRS  training  program  at  VP-31  in 
January  1979.  The  system  was  designed  with  the  intention  of 
covering  every  facet  of  VP  Antisubmarine  Warfare  Training 
required  to  make  optimum  use  of  the  P-3  aircraft.  A  quality 
control  plan  was  included  within  the  system  to  validate  all 
aspects  of  the  instructional  design  and  to  provide  the 
necessary  means  for  program  revision  when  appropriate.  The 
design  of  the  evaluation  plan  was  unavoidably  restricted  by 
constraints  stemming  from  inadequate  assets.  Additionally, 
when  the  training  system  was  implemented,  the  available  funds 
and  manpower  were  insufficient  to  allow  the  inclusion  of  the 
evaluation  phase  of  the  system.  Thus,  not  only  was  the 
evaluation  plan  incomplete  according  to  classical  ISD 
standards,  but  also  the  plan,  as  it  existed,  was  never 
formally  implemented  into  the  training  system. 

Another  serious  problem  plaguing  the  Quality  Control  plan 
was  a  material/media  revision.  Shortly  after  the  training 
system  was  implemented  in  1979,  it  was  discovered  that  due  to 
such  discrepancies  as  instructional  design  errors,   technical 
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content  changes,  and  editorial  errors,  some  of  the  designed 
materials  would  have  to  be  immediately  revised.  These 
revisions  included  a  change  in  format  for  tests.  The 
necessary  changes  were  immediately  incorporated  into  the 
system  without  being  validated.  This  violated  the  systematic 
basis  for  the  instructional  design  process  and  seriously 
jeopardized  the  evaluation  plan,  especially  in  the  area  of 
internal  evaluation. 

The  P-3  Instructional  System,  as  it  exists  today,  appears 
to  be  superior  to  the  traditional  system,  but  the  only  way 
this  can  be  verified  is  through  an  appropriate  evaluation 
plan.  Without  a  procedurally  correct  evaluation  plan  in 
effect,  the  ability  to  determine  whether  system  objectives 
are  being  met  and  graduates  are  capable  of  satisfactory 
performance  on  the  job  on  a  continuous  basis,  is  questionable 
at  best.  The  currently  used  method  of  evaluation  is  informal 
in  nature,  and  for  the  most  part,  consists  of  sporadic  usage 
of  a  traditionally  constructed  student  critique/questionnaire 
and  word-of -mouth  reports  at  monthly  conferences  held  by 
Commanding  Officers  of  Fleet  Squadrons. 

Initial  justification  for  the  redesign  of  VP  FRS  training 
as  stated  in  Chapter  One  was  to  provide  a  high-quality 
training  program  that  could  readilly  adapt  to  new  and  ad- 
vanced technology  while  reducing  operating  costs  and  manpower 
needs.    To  meet  these  requirements  a  new  instructional  pro- 
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gram  was  designed  using  the  ISD  process,  which  is  a  closed- 
loop  system.  The  evaluation  phase  of  this  type  of  program  is 
required  to  provide  systematic  maintenance  and  revision. 
Without  proper  evaluation,  there  cannot  be  a  closed-loop 
system,  but  only  a  rigidly  structured  traditional-type 
training  program.  Theoretically  one  can  say  that  without  an 
evaluation  phase,  there  is  no  such  thing  as  an  ISD  program. 

To  correct  the  Quality  Control  Plan  shortcomings 
mentioned  above,  a  problem  that  must  be  addressed  is,  how  can 
a  complete  evaluation  phase  be  incorporated  into  the  P-3 
Instructional  System  to  regain  the  benefits  of  the  powerful 
ISD  concept  under  which  it  was  designed? 

The  intent  of  this  thesis  is  to  make  contributions 
towards  a  more  effective  VP  FRS  training  system. 
Recommendations  will  be  presented  for  the  completion  and 
incorporation  of  the  P-3  evaluation  process,  which  will  aid 
in  a  better  overall  training  program. 

B.   THE  IMPORTANCE  OF  INSTRUCTIONAL  SYSTEM  EVALUATION 

The  undisputed  need  for  measurement  of  training 
effectiveness  has  long  been  realized.  Without  the  capability 
of  determining  how  well  the  instructional  program  is  being 
designed,  if  students  are  meeting  design  objectives,  and 
whether  or  not  students  are  properly  trained  for,  and  are 
capable  of,   adequately  doing  their  job   in   the   fleet,   a 
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training  system  can  only  be  mediocre  at  best.  According  to 
Hall,  Rankin,  and  Aagard  [Ref.  14],  a  major  problem  plaguing 
traditional  training  systems,  (i.e.,  non  ISD  systems)  is  the 
determination  of  training  effectiveness  largely  by  rational 
assessment  and  intuitions  of  personnel  intimately  involved  in 
the  training  process.  They  further  state  that  this  type  of 
information  tends  to  be  biased  and  lacks  the  detail  necessary 
for  improving  specific  aspects  of  training.  Therefore,  to 
determine  training  effectiveness  and  obtain  information 
suitable  for  proper  training  quality  control,  one  must  seek 
more  objective  and  systematic  means.  The  ISD  process  of 
instructional  development  employs  these  concepts  for  training 
system  evaluation. 

The  remainder  of  this  Chapter  will  cover  the  basics  of 
training  system  evaluation  followed  by  an  overview  of  the 
evaluation  methods  utilized  in  the  IPISD  model.  Finally,  the 
steps  utilized  in  the  quality  control  phase  of  the  P-3 
Instructional  System  will  be  discussed. 

1.   The  Nature  of  Instructional  System  Evaluation 

Evaluation  of  an  instructional  system  can  best  be 
described  as  a  means  of  analyzing  and  interpreting  system 
evidence  that  has  been  systematicaly  gathered.  The  basis  for 
training  program  evaluation  begins  with  a  consideration  of 
criteria  and  criterion  measures.  When  dealing  with  training 
programs,   two  classes   of  criteria,   internal  and  external 
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measures,  must  be  considered.  Internal  criteria  are  measures 
concerned  with  an  individual's  performance  in  the  training 
situation  such  as  objective  exams,  and  questionnaires 
reflecting  trainee  attitude  changes.  On  the  other  hand, 
external  criteria  are  utilized  to  assess  performance  on  the 
job  for  which  the  training  program  was  designed.  Measures  of 
quantity  and  quality  of  production  are  examples  of  external 
criteria. 

According  to  Landry  and  Trumbo  [Ref.  15],  Kirkpatrick 
[Ref.  16]  identified  four  "levels"  of  criteria  for  the 
evaluation  of  training  programs:  reaction,  learning, 
behavior,  and  results.  The  reaction  and  learning  levels  are 
internal  criteria.  Reaction  criteria  deals  with  how  the 
student  feels  about  the  training  program  and  learning 
criteria  attempts  to  measure  how  much  was  learned  as  a  result 
of  the  training.  The  last  two  levels  are  external  criteria. 
Behavioral  criteria  measures  performance  on  the  job,  while 
results  criteria  determines  the  utility  of  the  instruction 
with  respect  to  organizational  objectives. 

When  building  the  framework  for  training  system 
evaluation,  one  must  carefully  strive  for  the  proper  balance 
of  internal  and  external  criteria.  The  careful  measurement 
of  internal  criteria  can  provide  valuable  information,  but 
these  measurements  cannot  tell  what  impact  the  training  has 
had  on  job  behaviours  or  organizational  goals.   Evaluation  of 


39 


the  payoff  resulting  from  a  training  system  is  determined  not 
from  what  is  learned  in  training  but  how  the  individual 
performs  on  the  job. 

While  describing  the  principles  of  instructional 
design,  Gagne'  and  Briggs  [Ref.  1]  present  three  specific 
questions  concerning  instructional  systems  that  should  be 
answered  as  a  result  of  the  designed  evaluation  process:  (1) 
how  well  have  the  stated  objectives  of  instruction  been  met, 
(2)  is  the  new  program  better  than  the  one  it  is  replacing, 
and  (3)  does  the  new  program  produce  any  additional  effects? 

Evaluation  of  topics,  courses  and  instructional 
systems  is  carried  out  in  two  stages.  Stage  one,  formative 
evaluation,  is  undertaken  while  the  new  unit  is  being 
developed.  Its  purpose  is  to  provide  evidence  for  use  in 
making  revisions  and  improvements.  This  stage  is  often 
called  system  validation.  Stage  two,  summative  evaluation, 
is  concerned  with  the  measurement  of  student  performance  once 
the  course  or  program  has  been  developed.  In  other  words,  it 
is  primarily  concerned  with  instructional  system 
effectiveness.  The  focus  of  this  report  will  be  on 
summative- type  evaluation. 

The  basic  ISD  structured  training  models  used  by  the 
Armed  Services  makes  provision  for  summative  evaluation  to  be 
conducted  in  two  parts.    Part  one,  internal  evaluation,  is  a 
continuous    process   designed   to   check   instructional 
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effectiveness  (i.e.,  are  the  students  obtaining  course 
objectives?) .  The  primary  measure  of  effectiveness  is  made 
via  objective-reference  achievement  tests.  Other  measures 
include  time  to  complete  lessons,  attrition  rates,  and  the 
opinions  of  instructors  and  students  concerning  the  training. 
Part  two,  external  evaluation,  is  designed  to  measure  the 
performance  of  the  graduate  on  the  job.  The  purpose  of  this 
type  of  evaluation  is  to  discover  deficiencies  in  a 
graduate's  ability  to  perform  certain  required  tasks  when 
they  reach  the  job.  Once  the  causes  of  the  deficiencies  are 
determined,  revisions  to  the  system  can  be  made  to  correct 
the  flaws.  External  evaluation  also  ensures  that  the 
information  obtained  from  the  training  program  is  the 
information  needed  to  do  the  job.  The  two  most  accurate 
methods  of  external  evaluation,  testing  of  graduates  on  the 
job  and  direct  observation,  are  usually  too  costly; 
therefore,  more  often  than  not  evaluators  must  rely  on 
supervisors'  summary  evaluation  of  performance, 
questionnaires  and  interviews. 

The  most  important  points  concerning  the  conduct  of 
an  external   evaluation  are  first,   they  must  be  extremely 
specific  (task  level)  to  isolate  inadequate  performance,   and 
second,   it  must   take  place  within  a  few  months   after   the 
graduate  has  reached  the  job  so  that  the  skills  and  knowledge 
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acquired  on   the   job  will  not   influence   the   information 
received  from  the  training  program. 

2.   The  IPISD  Model  Evaluation  Process 

The  Interservice  Procedures  for  Instructional  Systems 
Development  Manual  [Ref.  8]  lists  course  evaluation  as  the 
final  phase  of  instructional  design.  This  important  step 
represents  the  beginning  of  a  perpetual  training  system 
evaluation  process  that  is  intended  to  make  certain  that  the 
training  program  continues  to  be  effective  and  to  provide  a 
student  output  quality  level  that  satisfies  fleet 
requirements. 

The  IPISD  model  evaluation  phase  capitalizes  on  the 
two  aspect  evaluation  processes  (internal  and  external) 
described  in  the  last  section.  The  internal  evaluation 
program  is  used  to  determine  if  a  course  has  been 
developed/conducted  according  to  the  "standards"  specified  in 
the  ISD  procedures.  Thus,  the  internal  quality  control 
aspect  seeks  to  answer  the  question,  "Is  the  course  teaching 
its  stated  objectives?".  The  IPISD  model  specifically  states 
that  the  effectiveness  of  both  the  instruction  and  the  ISD 
process  itself  will  be  evaluated. 

The  data  gathered  for  the  internal  evaluation  process 
deals  primarily  with  student  achievement  (i.e.,  attainment  of 
the  objectives,  as  represented  by  test  scores).  The  model 
also   recommends  collecting   information   such   as   trainee 
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background,  entry  skills,  trainee's  evaluation  of  methods  and 
media,  time  to  complete  lessons  and  instructor's  evaluation 
of  the  content.  This  kind  of  information  will  permit 
identification  of  weak,  ineffective,  impractical  or 
unpalatable  instruction,  and  thereby  lead  to  the  necessary 
revisions  in  the  instructional  program. 

The  external  evaluation  program  determines  the 
adequacy  of  the  instructional  design  and  development  process, 
and  trainee  proficiency,  by  a  standard  external  to  the 
course.  This  standard  is  normally  the  performance  of  the 
graduate  on  the  job.  The  external  evaluation  process 
considers  the  questions,  can  the  graduate  do  the  tasks 
required  on  the  job,  and  has  the  job  changed  since  it  was 
last  analyzed?  Data  gathered  for  the  evaluation  may  be 
obtained  from  graduates  and/or  their  supervisors.  In  an 
overview  of  the  IPISD  Model  external  evaluation  process, 
NAVEDTRA  106A  [Ref.  8]  stipulates  that  information  that  can 
be  obtained  from  course  graduates  includes  their  opinions 
about:  how  well  they  believe  they  can  perform  on  the  job, 
training  received  since  arriving  on  the  job,  how  well  the 
course  prepared  them  for  the  job,  the  parts  of  the  training 
which  were  relevant  to  the  job,  and  the  job-related  tasks 
which  caused  them  the  most  problems.  Additionally,  they 
state  that  the  information  that  may  be  gained  from 
supervisors  includes:   how  well  the  graduates  can  do  the  job, 
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how  those  graduates  measure  up  to  those  who  received  a 
different  form  of  training,  and  areas  in  which  the  graduates 
have  not  been  properly  trained. 

The  IPISD  model  recommends  the  administration  of  Job 
Performance  Measures  (JPMs)  in  the  field  30-90  days  after 
graduation  as  the  means  of  determining  whether  or  not 
graduates  can  perform  their  required  job  tasks.  Job 
performance  measures  are  "tests  that  are  used  to  evaluate 
proficiency  of  a  job  holder  on  each  task  he  performs."  [Ref. 
8]  The  Model  states  that  other  methods  such  as  interviews 
and  questionnaires  may  be  used,  but  that  these  techniques 
usually  inject  a  certain  amount  of  bias  into  the  evaluation 
process. 

Determining  whether  or  not  a  job  has  changed  since  it 
was  last  analyzed  is  a  matter  of  reviewing  the  present 
relationship  between  the  Job  Performance  Measures  and  actual 
job  requirements.  Vineberg  and  Joyner  [Ref.  7]  describe  this 
process  as  determining  the  predictive  validity  of  the  Job 
Performance  Measures. 

Most  prominent  instructional  designers  seem  to 
consider  the  administration  of  Job  Performance  Measures  to 
graduates  as  the  most  reliable  way  to  measure  the  adequacy  of 
instructional  design.  This  method  provides  the  most  accurate 
means  of  isolating  the  causes  of  performance  discrepancies 
during  the  evaluation,   but  it  should  be  pointed  out  that   it 
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is  also  the  most  expensive  to  use.  The  bottom  line  in 
external  evaluation  is  summed  up  very  explicitly  in  the  IPISD 
model  caution:  don't  revise  simply  for  the  sake  of  revision. 
When  the  majority  of  graduates  and  supervisors  are  satisfied 
with  the  quality  of  training,  few,  if  any,  system  changes  are 
usually  needed. 

3.   The  P-3  Instructional  System  Evaluation  Process 

The  final  phase  of  the  P-3  Instructional  System  is 
dedicated  to  quality  control.  The  purpose  of  the  Quality 
Control  program  is  to  ensure  that  the  system  is  meeting  its 
goals  in  the  most  optimum  manner  possible.  The  program's 
general  framework  is  composed  of  two  primary  parts.  Part 
one,  quality  control  of  materials,  aids  in  the  assurance  that 
all  applicable  materials  are  up-to-date  and  technically 
correct.  Part  two,  quality  control  of  the  instructional 
system  itself,  aids  in  the  assurance  that  the  system  is 
functioning  effectively. 

Generally  speaking,  the  quality  control  program's 
basic  guidelines  specifies  that  material  revisions  will  be 
made  when  the  performance  data  gathered  from  device  sessions 
indicate  a  need  to  revise  the  written  material  that  teach  the 
performance  and  when  data  support  the  existence  of  a  content 
or  instructional  deficiency.  Data  gathered  for  these 
evaluation  purposes  may  be  obtained  from  instructor  or 
student  inputs,   analysis  of  student  performance  and  attitude 
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data,  and  the  Naval  Air  Training  and  Operating  Procedures 
Standardization  (NATOPS)  manual  update  process. 

Part  two,  quality  control  of  the  P-3  Instructional 
System  itself,  is  a  continuous  process  devised  to  identify 
and  resolve  problems  within  different  aspects  of  the  system. 
System  revisions,  to  correct  problems,  are  designed  to  be 
made  without  deteriorating  other  parts  of  the  system.  Thus, 
any  problem  should  upgrade  the  entire  system  and  not  correct 
one  problem  at  the  expense  of  creating  new  ones  elsewhere  in 
the  system.  Revisions  to  the  system  may  originate  from 
changes  in  the  Master  Course  Syllabus  (the  primary  scheduling 
tool  for  the  Instructional  System) ,  in  resource  requirements, 
in  management  procedures,  or  in  course  content.  Strict 
documentation  and  distribution  of  revisions  are  extremely 
important  in  this  instructional  system,  since  the  system  is 
large  and  the  two  units  using  it  are  geographical'/  separated. 

This,  of  course,  has  been  a  very  broad  overview  of 
the  quality  control  phase  of  the  P-3  Instructional  System 
The  entire  phase  is  very  detailed  and  complex  as  one  would 
expect  from  such  a  large  training  program.  A  more  thorough 
discussion  providing  all  of  the  necessary  details  will  be 
presented  in  Chapters  Three  and  Five. 
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C.   SCOPE  OF  THESIS 

When  one  strictly  adheres  to  ISD  methodology,  internal 
evaluation  (i.e.,  evaluation  to  determine  the  effectiveness 
of  the  performance  of  an  instructional  product)  and  external 
evaluation  (i.e.,  evaluation  of  the  training  product  in  the 
operating  environment)  are  extremely  complex  tools  to  design, 
implement  and  operate.  Each  procedure  of  an  ISD  based 
evaluation  plan  is  related  to  the  entire  instructional  system 
in  such  a  manner  that  if  a  nonvalidated  change  is  made  in  the 
system,  the  evaluation  plan  can  be  seriously  degraded.  This 
appears  to  be  especially  true  in  the  area  of  internal 
evaluation.  For  example,  the  media/materials  revision  of  the 
P-3  Instructional  System  that  took  place  in  1979  has  made  it 
necessary  for  numerous  design  changes  to  be  made  to  the 
Quality  Control  Plan  before  an  accurate  internal  evaluation 
process  can  exist.  Due  to  the  extent  of  these  required 
design  changes,  an  analysis  of  the  entire  P-3  Instructional 
System  Quality  Control  Plan  (i.e.,  internal  and  external 
evaluation)  is  beyond  the  scope  of  this  thesis.  Since  the 
question  that  needs  an  immediate  answer  (i.e.,  the  bottom 
line  in  VP  training)  is,  can  the  VP  FRS  graduates  perform 
satisf actorilly  on  the  job,  the  scope  of  this  report  will  be 
limited  to  an  analysis  of  the  external  evaluation  process. 
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III.   P-3  INSTRUCTIONAL  SYSTEM  EXTERNAL  EVALUATION  SPECIFICS 

The  instructional  system  that  is  designed  according  to  an 
ISD  model  is  constructed  as  a  complete  cycle.  The  evaluation 
phase  is  the  key  element  in  that  it  provides  feedback 
necessary  to  complete  the  cycle.  This  feedback  is  the  means 
of  ensuring  continuing  quality  in  graduate  performance. 
Evaluation  is  that  integral  part  of  the  training  system  with 
the  capability  to  locate  discrepancies  and  initiate  system 
changes.  An  evaluation  process  is  not  a  general  item  that  is 
used  to  "check"  a  training  system,  but  rather  a  specific  tool 
that  must  be  designed  for  each  individual  system. 

The  evaluation  process  designed  by  Courseware,  Inc.,  for 
incorporation  into  the  FRS  Training  System  is  titled  the 
P-3  Instructional  System  Quality  Control  Plan.  As  stated  in 
the  last  chapter,  the  plan  was  designed  under  constraints 
that  prevented  the  development  of  complete  procedures 
necessary  for  evaluation  of  the  system.  The  resulting 
shortcomings  in  the  plan  and  their  possible  remedies  will  be 
discussed  in  Chapter  Five  of  this  report. 

Chapter  Two  presented,  in  part,  a  general  overview  of 
both  the  IPISD  model  evaluation  procedures  and  the  P-3 
Instructional  System  evaluation  procedures  for  background 
information.    This   chapter   will  focus  on   the   details  of 


48 


evaluating  the  P-3  Instructional  System,  especially  in  the 
area  of  external  evaluation.  The  material  that  is  presented 
is  a  summary  of  the  P-3  Instructional  System  Quality  Control 
Plan  [Ref .  17]  and  Management  Plan  [Ref .  18]  and  is  intended 
to  provide  the  reader  with  enough  detail  to  follow  the 
analysis  of  the  plan  in  Chapter  Five.  The  plan  is  presented 
just  as  it  was  designed  by  Courseware,  Inc.  The  first  three 
sections  of  this  chapter  provide  general  information  that 
pertains  to  the  entire  quality  control  plan  (i.e.,  internal 
and  external  evaluation) ,  while  the  remaining  two  sections 
will  cover  specifics  of  the  external  evaluation  plan. 

A.   PURPOSE  OF  THE  QUALITY  CONTROL  PLAN 

Quality  control  provides  the  means  for  instructional 
personnel  to  systematically  ask  questions  designed  to 
evaluate  the  training  program.  The  devised  plan  to  conduct 
quality  control  involves  three  major  activities:  data 
gathering,  analysis  and  revision.  The  constraints  placed  upon 
data  gathering  produce  a  chain  reaction  resulting  in  similar 
limits  being  placed  upon  the  level  of  analysis  that  can  be 
accomplished  and  finally  dictating  the  number  and  kinds  of 
revisions  that  can  be  made  to  this  program. 

Three  distinctly  separate  stages  occur  in  the  quality 
control  effort.  The  first  stage,  validation,  takes  place 
during  the  actual  development  of  the  instructional  program, 
and   involves   trying  out  designed  material  with   individual 
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and/or  small  groups.  This  stage  allows  instructional  designs 
to  guarantee  specified  results.  The  second  stage,  internal 
quality  control,  is  a  continuous  process  that  takes  place 
during  the  implementation  of  the  training  program.  This 
stage  provides  for  the  identification  and  correction  of 
deficiencies  in  the  total  training  program  (i.e.,  both 
management  and  materials) . 

The  third  stage,  external  quality  control,  takes  place 
several  months  after  course  graduates  depart  for  fleet 
assignments.  This  process  attempts  to  identify  and  correct 
problems  pertaining  to  the  areas  of  course  relevance  to  real- 
world  operational  requirements.  Internal  and  external 
quality  control  must  be  conducted  continuously  to 
satisfactorily  maintain  the  training  program.  Figure  3.1 
details  the  entire  P-3  Instructional  System  quality  control 
process  and  its  outputs.  Note  the  connecting  nature  of  the 
process  that  links  all  three  stages  together. 

B.   SCOPE  OF  THE  QUALITY  CONTROL  PLAN 

This  quality  control  plan  was  designed  to  cover  the 
personnel  and  procedures  required  during  the  internal  and 
external  quality  control  stages.  During  internal  quality 
control,  data  gathering  and  analysis  will  be  directed  towards 
answering  the  question,  "is  the  training  program  effectively 
meeting    its   defined    instructional   and    performance 
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abj arrives?"  This  phase  cf  the  quality  control  process  is 
primarily  concerned  with  whether  or  no-  students  are  meeti-.  - 

established  minimum  performance  requirements. 

The  external  quality  control  portion  of  the  plan  will 
direct   its  data  gathering  and  analysis  toward  answering   the 

question,  "is  the  training  prccram  focusing  en  s<ills  and 
knowledge  required  in  the  real  operational  world?-  This 
process  addresses  two  <mds  of  discrepancies:  1  errors 
associated  with  initial  tas<  listing  or  tas<  selection 
procedures,  and  2      dated  concepts  and  procedures. 

Due  to  imposed  constraints  such  as  tine,  manpower,  ind 
money,  this  quality  control  plan  dees  not  address  the 
following : 

(1)  Procedures  for  quality  control  of  the  instruc- 
tional management  system;  mis  area  is  covered  in  Appendix 
1   of  the  ?-3  Instructional  System  Management  Plan. 

(2)  Zcmplete  procedures  for  the  external  quality  con- 
trol plan;  due  to  the  time  requirements  for  data  collection 
and  analysis,  the  timeframe  for  completion  of  the  external 
quality   pontrol   process  would  nave   extended   beyond   i.-.e 

scope  cf  the  contract  for  development  of  the  ?-2 
Instructional  System. 

2  Continuous  evaluation  and  revision  procedures; 
these  procedures  are  test  identified  reward  the  end  of  the 
internal  quality  control  process  on  the  oasis  of  a  needs 
assessment.    The   needs   assessment  and  identification   of 

maintenance  ongoing  revision  c  r  a c  e  i  - r  e  s  -'ere  intended  :z  re 
a  joint  Courseware  Navy  endeavor  cased  on  the  experience  of 
the  initial  quality  control  plan. 
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C.   CONSTRAINTS  OF  THE  QUALITY  CONTROL  PLAN 

The  extent  of  the  evaluation  (i.e.,  the  total  number  and 
kind  of  evaluation  questions  that  can  be  addressed  during 
internal  and  external  quality  control)  is  dependent  upon 
existing  priorities  or  client  constraints. 

1.  Design  Accomodations 

The  quality  control  plan  was  custom  designed   to 
accomodate  the  following  restrictions: 

(1)  Fixed  Naval  personnel  resources:  this  limitation 
affects  the  amount  and  kind  of  data  collection  activities 
during  internal  quality  control. 

(2)  Fixed  trainer  assets:  this  limitation  imposes 
strict  enforcement  of  scheduling  which  limits  the  time 
available  for  student/instructor  interview  data  collection. 

(3)  Fixed  course  length:  this  limitation  was  appli- 
cable during  the  implementation  year.  It  didn't  allow 
enough  flexibility  in  training  time  to  make  up  for  initial 
instructional  or  system  deficiencies. 

(4)  Fixed  number  of  students  within  a  fixed  contract 
delivery  schedule:  this  restriction  may  result  in  a  non- 
representative  target  population  sample  size. 
Additionally,  initial  training  program  effectiveness  may  be 
degraded  by  an  insufficient  amount  of  trainer  assets. 

(5)  Training  accountability:  during  the  implementa- 
tion year  this  restriction  forces  the  emphasis  to  be  placed 
on  training  rather  than  data  collection  for  analysis  and 
revision  of  the  instructional  materials. 

(6)  Learning  environment  limitations:  lack  of  readi- 
ness in  this  area  restricted  the  evaluation  of  the 
instructional  materials  and  the  efficiency  of  the 
instructional  management  system. 

2.  Revision  Priorities 

Based   upon   the   above   considerations,    several 
revision  priorities  were  established  for   internal   quality 
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control.  Since  the  primary  focus  of  this  report  is  on  the 
external  evaluation  phase,  the  priorities  will  not  be 
covered,  but  may  be  found  in  their  entirety  in  the  P-3 
Instructional  System  Quality  Control  Plan.  Revisions  made  to 
the  training  program  will  be  made  according  to  the  kind  and 
amount  of  data  collected. 

D.   DESIGN  OF  THE  EXTERNAL  QUALITY  CONTROL  PLAN 

This  section  of  the  Quality  Control  Plan  discusses  the 
general  procedures  to  be  used  in  conducting  the  development, 
data  collection,  analysis,  and  revision  phases  for  external 
evaluation.  Further  procedural  detail  was  intended  to  be 
provided  through  a  joint  Navy /Courseware  endeavor  upon 
completion  of  the  second  class  of  students  for  internal 
quality  control. 

1.   Development  Phase 

The  purpose  of  the  development  phase  is  to  provide 
data  collection  instruments  for  use  in  the  data  collection 
phase.  When  considering  analysis,  the  following  questions 
apply: 

(1)  Can  the   initial  course   graduate   perform  at  the 
level  expected  while  on  the  job? 

(2)  Has  the   initial  course   graduates'   training  been 
too  extensive  in  any  skill  area? 

(3)  Has  the  initial  course  graduate  attained  the 
proper  knowledge  and  skills  upon  arrival  at  the  operational 
squadron? 
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(4)  Does  the  initial  course  graduate  feel  the  training 
program  adequately  prepared  him  for  the  real-world  job 
tasks? 

Attempts  to  answer  these  questions  will  be  conducted 
through  two  types  of  data  collection  instruments.  The  first 
type,  questionnaires,  will  be  sent  to  squadron  supervisors  of 
initial  course  graduates  and  course  graduates  themselves.  The 
second  type,  interview  questions,  will  be  administered  on- 
site  to  both  supervisors  and  course  graduates. 
2.   Data  Collection  Phase 

This  phase  consists  of  two  major  steps: 

(1)  Questionnaire  data  gathering  activities. 

(2)  On-site  interviews  with  supervisors  and  initial 
course  graduates  who  have  been  on  the  job  approximately  six 
months. 

Questionnaires   will   be  sent  to  all   initial  course 

graduates   and   their  job  supervisors.    For  this   particular 

quality  control  plan,   four  questionnaires  currently  exist. 

Appendix  A  shows  a  sample  initial  course   graduate   follow-up 

questionnaire,   while  Appendix  B  shows  a   specific   sample 

course   graduate   questionnaire   for   the   pilot   student. 

Appendix   C  displays   a  sample   job  supervisor    follow-up 

questionnaire   and  Appendix  D  shows  a   specific   sample   job 

supervisor   follow-up  questionnaire  for  a  pilot   supervisor. 

As  specified  in  the  Management  Plan,  follow-up  questionnaires 

will   be   returned   to   the  VP-31   Instructional   System 

Department,   for  data  summarization  and  analysis,   by  course 
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Subject  Matter  Experts  (SMEs) .    On-site  interviews  will  be 
conducted  at  sites  selected  by  the  Navy. 

3.  Analysis  Phase 

This  phase  addresses  the  following  activities: 

(1)  Summarization  of  collected  data. 

(2)  Analysis  of  collected  data. 

(3)  Revision  procedures  on  the  basis  of  analysis. 

Discrepancies  in  task  skills  or  knowledges  will 
require  task  analysis  and  identification  of  learning  goals. 
Learning  goals  will  be  recorded  on  the  Revision  Specification 
Worksheet  (Appendix  E) ,  placed  in  a  learning  heirarchy  and 
assigned  for  authoring.  The  course  syllabus  will  be  modified 
as  necessary. 

The  Management  Plan  assigns  (SMEs)  the  responsibility 
for  summarizing  and  analyzing  supervisor  and  course  graduate 
questionnaire  responses.  The  SMEs'  duties  consist  of 
comparing  fleet  squadron  input  to  input  from  other  data 
sources.  They  then  determine  what  action  is  to  be  taken  to 
correct  discrepancies.  The  summarized/analyzed  data  is  then 
forwarded  to  the  training  officer/course  supervisor  for 
review.  The  Management  Plan  also  requires  a  few  specific 
steps  to  be  carried  out  in  the  analysis  data  stage.  These 
actions  are  discussed  in  Chapter  Five. 

4.  Revision  Phase 

The  revision  phase  is  the  responsibility  of  the  ISD 
Department  of  the  Navy.    The  instructional  materials  will  be 
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revised  in  accordance  with  revision  specifications  from  the 
analysis  phase.  According  to  the  Management  Plan,  revisions, 
for  the  most  part,  will  be  dictated  by  the  data  collection 
and  analysis  results.  SMEs  and  instructors  will  determine 
jointly,  the  nature  and  extent  of  the  revisions  to  be  made, 
but  SMEs  are  ultimately  responsible  for  instituting  the 
actual  changes.  A  few  specific  procedural  details  concerning 
revisions  are  presented  in  the  Management  Plan.  These 
procedures  will  be  discussed  in  Chapter  Five. 

E.   PERSONNEL  AND  SCHEDULING  REQUIREMENTS 

The  development  and  data  collection  phases  were  to  be 
initiated  by  Courseware,  Inc.,  and  due  to  the  time 
requirements  for  questionnaire  distribution  and  on-site 
interviewing,  the  analysis  and  revision  phases  were  to  be 
conducted  by  the  Navy. 
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IV.   METHODOLOGY 

The  data  used  in  this  thesis  came  from  Uniformed  Services 
publications,  commercial  books  and  reports,  and  various 
studies  conducted  by  and  for  the  Military  Services.  This 
data  reflects  the  current  design,  implementation  and 
operation  standards  for  United  States  Navy  training  programs. 
Additional  background  data  was  gathered  through  informal 
interviews  with  various  VP  FRS  training  administrators  and 
managers. 

The  intention  of  this  thesis  will  be  realized  through  the 
analysis  of  the  P-3  Instructional  System  External  Evaluation 
Plan.  The  plan  will  be  related  to  the  Interservice 
Procedures  for  Instructional  System  Development  model 
external  evaluation  process  as  well  as  other  standardized  ISD 
concepts.  Prom  this  analysis,  a  better  understanding  of  the 
external  evaluation  plan  will  be  gained,  and  recommendations 
for  an  improved  evaluation  process  and  overall  training 
system  will  be  made.  The  report  will  conclude  with  a 
discussion  on  the  feasibility  of  implementing  the  Quality 
Control  Plan. 
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V.   ANALYSIS  OF  THE  P-3  INSTRUCTIONAL  SYSTEM  EXTERNAL 

EVALUATION  PLAN 

Previous  sections  of  this  thesis  have  related  the 
importance  of  utilizing  the  ISD  process  to  develop  VP  FRS 
aircrew  training,  how  an  evaluation  process  is  essential  to 
the  creation  of  the  closed-loop  training  system,  and  the 
development  status  for  the  P-3  Instructional  System  Quality 
Control  Plan.  This  section  will  analyze  the  existing  P-3 
Instructional  System  External  Quality  Control  Plan.  The 
author  realizes  that  due  to  extenuating  circumstances  the 
plan,  as  it  now  stands,  is  not  complete,  continuous,  or 
incorporated  into  the  FRS  training  program.  This  analysis  is 
being  conducted  to  offer  suggestions  that  should  aid  in  the 
completion  and  implementation  of  the  plan;  thus  resulting  in 
an  improved  VP  FRS  aircrew  training  system.  This  analysis  of 
the  P-3  Instructional  System  External  Evaluation  Plan  is  not 
an  attempt  to  segregate  it  from  the  remainder  of  the  system, 
but  is  simply  an  attempt  to  focus  on  improving  one  aspect  of 
the  training  program  with  the  intention  of  strengthening  the 
entire  system. 

The  procedure  for  the  analysis  will  consist  of  comparing 
the  external  evaluation  phase  of  the  P-3  Instructional  System 
to  the  external  evaluation  phase  of  the  IPISD  model.  The 
IPISD  model  was  chosen  as  the  comparison  standard  because  it 
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provides  a  description  of  the  approved  techniques  and 
procedures  to  be  used  for  interservice  training  and  includes 
requirements  from  all  branches  of  the  Armed  Services. 
Additionally,  Vineberg  and  Joyner  [Ref.  7]  found,  in  a  study 
of  all  Armed  Forces  ISD  models,  that  following  external 
evaluation,  the  IPISD  model  provides  the  most  guidance  for 
isolating  causes  of  performance  discrepancies.  The  format 
for  the  comparison  will  consist  of  presenting  a  summary  of 
each  step  of  the  IPISD  Model  external  evaluation  procedures 
immediately  followed  by  a  discussion  concerning  the  P-3 
Instructional  System  External  Quality  Control  Plan 
accompli sments  for  that  step  and  suggested  recommendations 
for  changes  to  the  plan. 

The  section  will  conclude  with  a  discussion  of  the 
feasibility  of  implementing  the  external  evaluation  program, 
once  it  has  been  redesigned  and  can  meet  the  appropriate  ISD 
standards. 

A.   COMPARISON  OF  THE  EXTERNAL  EVALUATION  PROCESSES 
1.   Introduction 

The  IPISD  Model  specifies  that  the  primary  purpose 
for  conducting  external  evaluation  is  to  ensure  that  course 
graduates  can  do  the  job  for  which  they  were  trained,  at  the 
expected  level  of  competency.  External  evaluation  also 
checks   to   see   if  the  job  has  changed   since   it  was   last 
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analyzed,  thus,  in  a  sense,  providing  a  revalidation  of  the 
last  task  listing.  Figure  5.1  illustrates  the  position 
(Block  V.2)  in  the  ISD  cycle  where  external  evaluation  takes 
place.  One  must  understand  that  although  the  evaluation  is 
conducted  after  students  have  successfully  completed 
instruction  and  reported  to  the  job,  it  is  still  a  part  of 
the  closed-loop  process  required  in  an  ISD  training  program. 
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Source:   Ref .  8:  p.  63 
Figure  5.1.   IPISD  Model  Block  Diagram 

The   IPISD  Model  requires  the  following   sources  of 
input  data  to  conduct  external  evaluation: 

(1)  All  available  data  from  internal  evaluation. 

(2)  The  job  analysis,  task  selection,  job  performance 
measures,  and  setting  selection  information  developed  in 
Phase  One. 
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(3)   Any  documentation  approved  by  higher   authority 
which  will  change  the  requirements  of  the  course. 

Procedurally  correct  and  properly  conducted  external 
evaluation  will  either  confirm  that  course  graduates  can  do 
their  job  at  required  performance  levels  or  that  students 
cannot  meet  job  performance  standards  and  which  tasks  are 
causing  this  failure.  The  specific  steps  for  planning  and 
carrying  out  external  evaluation  are  shown  in  Figure  5.2. 

Discussion:  Basically  the  IPISD  Model  and  the  P-3 
Instructional  System  External  Quality  Control  Plan  (QCP)  seem 
to  agree  on  their  underlying  philosophies  for  conducting 
external  evaluation.  Both  models  are  primarily  concerned 
with  whether  or  not:  (1)  graduates  can  adequately  handle  the 
job  for  which  they  were  trained,  (2)  the  job  task 
requirements  have  changed,  and  (3)  course  updates  due  to 
material  revisions  can  be  made. 

The  QCP  states  in  its  introduction  that  specific 
procedures  for  external  quality  control  will  not  be  given  due 
to  contract  time  limitations.  Therefore,  an  initial  external 
evaluation  of  the  P-3  Instructional  System  was  never 
conducted  nor  were  procedures  identified  for  ongoing 
evaluation  and  revision.  It  should  be  reemphasized  that 
although  these  procedures  were  never  devised,  the  QCP  does 
concur  that  they  are  necessary. 
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The  QCP  design  was  hampered  by  several  constraints. 
Although  most  of  the  constraints  were  primarily  concerned 
with  either  the  total  initial  evaluation  effort,  or  internal 
evaluation,  one  constraint  had  a  serious  effect  on  the 
external  evaluation  program.  The  constraint  of  concern  is  a 
shortage  of  naval  personnel  resources.  This  lack  of  properly 
trained  personnel  imposes  serious  restrictions  on  successful 
evaluation  planning,  data  collection,  data  analysis,  and 
training  program  revision. 

Recommendations:  The  total  benefit  derived  from  an 
ISD  structured  training  program  cannot  be  realized  without 
adequate  personnel  capable  of  conducting  external  evaluation. 
Sufficient  quantities  of  knowledgeable  personnel  should  be 
formed  into  a  team  that  is  capable  of  providing  continuous 
evaluation.  Optimally  the  team  should  be  well  versed  in  the 
area  of  ISD  procedures  and  techniques,  should  have  a  good 
understanding  of  the  job  to  be  evaluated,  and  should  not  be 
connected  to  the  training  program.  One  possibility  for 
forming  such  a  team  is  to  employ  an  instructional 
psychologist  as  team  leader  and  use  IS  Department  SMEs/track 
managers  rotating  out  of  VP-30  and  VP-31  as  the  remaining 
members  to  form  an  independent  evaluation  unit.  Another 
means  of  forming  the  evaluation  team  could  be  through 
utilization  of  personnel  from  agencies  such  as  TAEG,  NTEC,  or 
Naval   Instructional  Technology  Development  Center   (NITDC) . 
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The  problem  with  this  sort  of  team  would  be  their   lack  of 
knowledge  concerning  the  job  which  they  are  evaluating. 

The   formation  of  an  independent  external   evaluation 
team  is  a  valuable  part  of  the  ISD  concept.    A  study   should 
be  conducted  immediately  to  determine  how  an  effective   and 
efficient  evaluation  team  can  be  formed. 
2.   Procedure 

The  procedures  for  conducting  external  evaluation  in 
accordance  with  the  IPISD  Model,  Figure  5.2,  are  divided  into 
three  major  sections:  planning  evaluation,  collecting  data, 
and  consolidating  the  data  and  making  recommendations.  These 
three  sections  will  be  covered  in  detail. 

a.   Plan  Evaluation 

Planning  evaluation  is  a  process  that  is  to  be 
completed  prior  to  beginning  an  external  evaluation.  During 
this  process  decisions  are  made  to  determine  how  the 
evaluation  will  be  conducted  by  addressing  four  questions: 
who  will  provide  data,  what  data  is  needed,  exactly  when  the 
external  evaluation  will  take  place,  and  what  methods  will  be 
used  to  gather  the  data?  Since  planning  the  evaluation  is 
such  an  important  part  of  the  quality  control  concept,  these 
four  questions  will  be  covered  separately. 

(1)  Determine  Who  Will  Provide  the  Data.  The 
IPISD  Model  specifies  that  in  most  cases  data  gathered  for 
analysis  will  come  from  five  major  sources: 
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(1)  Baseline  data  gathered  prior  to  the  development  of 
instruction  (see  Figure  5.1,  block  II. 2). 

(2)  On-the-job  course  graduates. 

(3)  Supervisors  of  on-the-job  course  graduates. 

(4)  Evaluation  team  members  (ideally  composed  of  com- 
mand and  school  personnel) . 

(5)  Records  documenting  students'   performance  during 
instruction. 

Additionally,  a  check  should  be  made  on 
students  who  are  not  assigned  to  the  job  for  which  they  are 
trained.  Find  out  if  they  were  initially  assigned  to  that 
job  and  why  they  have  been  reassigned. 

Discussion:  The  QCP  specifies  that  data 
will  be  collected  from  both  course  graduates  and  supervisors 
of  course  graduates.  However,  the  plan  doesn't  address 
utilizing  data  from  the  baseline  study,  evaluation  team 
members,  or  records  of  students'  performance.  These  three 
sources  of  information  can  provide  invaluable  information 
during  external  evaluation.  Baseline  data  becomes  a 
necessity  if  the  technique  of  job  performance  evaluation  is 
to  be  utilized  to  assess  graduates'  on-the-job  performance, 
and  evaluation  team  members'  input  data  is  extremely 
important  no  matter  what  technique  is  used  to  evaluate  a 
graduate's  ability  to  perform  on  the  job. 

The  construction  of  the  P-3  Instructional 
System  is  such  that  records  of  students'  performance  during 
instruction  are  available  for  use  in  evaluation.    This  sort 
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of   data  becomes  especially  important  when  the  other   sources 
of  data  provide  conflicting  conclusions. 

Recommendations:  The  QCP  should  state 
specifically  and  completely  who  will  provide  the  data  needed 
for  evaluation.  The  author  feels  that  the  IPISD  Model 
provides  complete  coverage  of  who  will  provide  data. 
Therefore,  the  QPC  would  do  well  to  adopt  the  data  collecting 
procedures  specified  by  the  IPISD  Model. 

(2)  Determine  What  Data  Are  Required.  The  IPISD 
Model  suggests  that  data  gathered  from  graduates,  supervisors 
of  graduates,  evaluation  teams,  and  students'  records  be 
compared  to  the  baseline  data  to  obtain  at  least  partial 
answers  to  questions  such  as: 

(1)  Do  graduates  of  the  current  training  program  show 
better  performance  on-the-job  than  graduates  of  past 
programs? 

(2)  Do  graduates  of  the  current  training  program  re- 
quire less  on-the-job  training  than  graduates  of  other 
courses? 

Additionally,   the   IPISD  Model  states   that 

the   data  gathered  from  the  four  sources  should  be   directed 

toward  answering  specific  questions.    The  data  gathered  from 

graduates  who  are  working  on-the-job  should  provide   answers 

to  such  questions  as: 

(1)  How  well  are  they  performing  on  the  job? 

(2)  What  is  the  extent  of  training  received  since 
their  arrival  on-the-job? 
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(3)  How  well  did  the  course  prepare  them  for  the  job? 

(4)  What  parts  of  the   training   program  were  relevant 
and  irrelevant  to  the  job? 

(5)  In  doing  the  job,  how  often  do  they  use  the  skills 
taught? 

(6)  In  doing  the  job,  what  tasks  do  they  find  the  most 
difficult? 

(7)  In  doing  the  job,  for  which  tasks  do  they  feel  the 
least  adequately  prepared? 

(3)   In  doing  the  job,   which  tasks  do  they  feel  they 
perform  the  best? 

(9)   How  do  they   think  the   training  program  could  be 
changed  to  better  prepare  them  for  the  job? 

From   supervisors  of  the  working  graduates, 

get  answers  to  such  questions  as: 

(1)  How  well  are  the  graduates  performing  on-the-job? 

(2)  How  do  the  graduates  compare  to  graduates  of  other 
courses,  or  those  who  received  no  training? 

(3)  What   type   and  how  much  on-the-job  training  have 
the  graduates  received? 

(4)  In  what  areas  are  the  graduates  most   adequately 
and  inadequately  prepared? 

(5)  In  what  ways  could  the   training  program  be  im- 
proved? 

(6)  Has  the  graduate  had  any  problems  operating  equip- 
ment? 

(7)  Has  the  graduate's  performance   been  well  above  or 
well  below  the  level  expected? 

(8)  Has  the  graduate  been  promoted  or   recommended  for 
promotion? 

The  evaluation  team  should  attempt   to  get 
answers  to  such  questions  as: 
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(1)  Did  the  graduate  score  well  on  the  job  performance 
measures? 

(2)  Did  the  graduate   fail  any  of  the  job  performance 
measures? 

(3)  How  close  do  the  job  performance  measures  fit  the 
actual  job  requirements? 

(4)  Were  guidelines   followed  to  ensure  that  perfor- 
mance tests  were  properly  administered  and  scored? 

(5)  Is  the  job  in  question   structured  in  accordance 
with  regulations  or  approved  doctrine? 

(6)  Does  the  supervisor  have  satisfactory  knowledge  of 
the  job? 

(7)  Is  any  other  information  available  that   should  be 
considered  in  making  revision  decisions? 

Discussion:  Data  gathering  is  the  key 
element  in  ISD  external  evaluation.  Therefore,  any  plan 
designed  to  conduct  external  evaluation  must  emphasize  the 
collection  of  adequate  amounts  of  data  that  are  relevant  to 
the  questions  for  which  answers  are  being  sought.  The  IPISD 
Model  states  that  ideally,  external  evaluation  will  be  based 
principally  upon  actual  job  performance  as  measured  by  JPMs 
that  were  produced  during  the  design  of  ISD  Model.  The  model 
goes  on  to  say  that  often  because  of  difficulties  some  data 
will  have  to  be  collected  in  another  way. 

The  P-3  Instructional  System  design  concepts 
did  not  allow  for  the  production  of  complete  JPMs  during  its 
development.  Therefore,  a  means  to  collect  hard  data  for 
graduate  on-the-job  performance  evaluation  does  not  presently 
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exist.  This  forces  the  QCP  to  rely  primarily  on 
questionnaires  and  interviews  to  gain  the  data  necessary  to 
answer  the  questions  proposed  above  by  the  IPISD  Model. 

Recommendations:  The  abscence  of  JPMs  in  a 
military  service  ISD  based  training  program  is  not  unusual. 
The  advanced  techniques,  time,  and  manpower  required  to 
construct  these  measures  often  cause  them  to  be  omitted 
during  the  training  program  development.  Additionally, 
utilization  of  the  JPMs  to  gain  accurate  performance  data  for 
evaluation  might  be  extremely  desirable,  but  it  is  usually  so 
expensive  that  it  is  not  feasible. 

Well-designed  questionnaires  and  properly 
conducted  interviews  are  capable  of  producing  a  wealth  of 
information.  It  is  recommended  that  if  it  is  not  feasible  to 
construct  JPMs  at  the  present  time,  the  QCP  be  structured  to 
make  optimum  use  of  these  alternate  data  gathering  tools. 
Also  provisions  should  be  made  to  allow  the  restructuring  of 
the  IPISD  model  proposed  questions  that  deal  with  JPMs. 

(3)  Determine  When  External  Evaluation  will  Take 
Place .  Generally  speaking,  a  course  graduate  should  be  given 
at  least  30  days  on  the  job  before  he  is  contacted.  This 
period  of  time  will  allow  him  to  become  accustomed  to  his  new 
job  and  will  give  his  supervisor  encugh  time  to  observe  him 
and  provide  a  useful  evaluation  of  his  performance. 
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Do  not  allow  more  than  90  days  to  pass 
before  contacting  the  graduate  and  his  supervisor.  Exceeding 
this  length  of  time  will  make  it  difficult  for  the  graduate 
to  remember  the  particulars  of  his  instruction  and  how  well 
it  prepared  him  for  the  job.  Also,  it  will  hamper  the 
supervisor's  ability  to  recall  how  well  the  graduate 
performed  during  the  first  few  weeks  on  the  job. 

Discussion:  The  QCP  specifies  that 
interviews  be  conducted  with  initial  course  graduates  and 
their  supervisors  approximately  six  months  after  the  graduate 
begins  his  operational  assignment.  The  plan  also  requires 
that  questionnaires  be  distributed  and  retrieved,  but  gives 
no  timeframe  for  this  to  be  accomplished.  The  author  assumes 
that  the  questionnaires  were  to  be  distributed  during  the 
same  timeframe  that  the  interviews  were  conducted  (i.e., 
approximately  six  months  after  the  graduate  begins  his  job) . 
Furthermore,  since  the  plan  only  addresses  interview  times 
for  initial  course  graduates,  one  must  assume  that  the  same 
relative  timeframe  would  apply  to  graduates  on  a  continuous 
basis.  The  general  assumption  that  is  being  made  is  that 
the  QCP,  as  it  now  stands,  calls  for  external  evaluation  to 
take  place  approximately  six  months  after  a  graduate  begins 
his  job  in  the  fleet. 
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The  six-month  wait,  before  administering 
external  evaluation  as  required  by  the  QCP,  certainly  satis- 
fies the  IPISD  Model's  minimum  delay  of  30  days.  This  delay 
allows  the  graduate  time  to  get  comfortable  in  the  job  and 
gives  the  supervisor  a  reasonable  amount  of  time  to  observe 
him.  On  the  other  hand,  the  six-month  wait  far  exceeds  the 
IPISD  Model's  recommendation  for  a  maximum  delay  of  three 
months.  This  excessive  delay  would  certainly  make  it  diffi- 
cult for  the  graduate  to  recall  details  of  his  instruction 
and  for  his  supervisor  to  pinpoint  the  graduates'  initial 
capabilities.  Additionally,  the  author  feels  that  a  six- 
month  delay  would  impose  a  severe  bias  on  the  evaluation  due 
to  the  strong  influence  of  squadron  training  programs. 
During  the  author's  three  years  in  a  VP  fleet  squadron,  it 
was  noted  that  except  for  rare  instances,  squadron  training 
for  a  particular  job  began  immediately  upon  assuming  the 
duties  of  that  job.  Thus,  after  a  six-month  wait,  a 
graduate's  performance  on  the  job  would  be  clouded  by  six 
months  of  intensive  squadron  training. 

Recommendations:  External  evaluation  should 
be  scheduled  to  take  place  during  a  timeframe  that  will 
minimize  the  degradations  referred  to  in  the  discussion. 
Based  upon  the  above  information,  it  is  suggested  that  the 
QCP  adopt  the  IPISD  Model's  recommendation  of  conducting 
external   evaluation   between   30  days  and  90   days   after   a 
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graduate  reports  to  his  fleet  job.  Due  to  the  time  normally 
required  for  one  to  become  "settled"  into  a  new  job,  the 
optimum  timeframe  for  the  evaluation  would  probably  be  60-90 
days  after  the  graduate  begins  his  job. 

(4)   Determine  How  the  Data  Will  be  Gathered 

The  three  primary  methods  of  gathering 
external  evaluation  data  are: 

(1)  Through  job  performance  measures. 

(2)  Through  questionnaires. 

(3)  Through  personal  interviews. 

The  basic  procedure  is  to  provide  question- 
naires to  the  maximum  number  of  graduates  and  their  supervi- 
sors and  to  pick  a  random  sample  of  graduates  and  their 
supervisors  in  the  field  for  personal  interviews.  When  a  job 
performance  evaluation  is  possible,  it  is  usually  subject 
matter/evaluation  experts  who  collect  data  from  job  perfor- 
mance measures  and  actual  observation  of  the  graduate  on  the 
job. 

Discussion:  The  QCP  provides  for  external 
evaluation  data  to  be  gathered  by  questionnaires  and  personal 
interviews,  which  are  two  of  the  three  methods  mentioned  in 
the  IPISD  Model.  However,  the  QCP  does  not  address  data 
gathering  via  job  performance  measures,  because  as  stated 
earlier,  complete  job  performance  measures  were  not  construc- 
ted during  the  development  of  the  P-3  Instructional  System. 
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Specific  procedures  for  determining  how 
external  evaluation  data  will  be  gathered  are  not  covered  in 
the  QCP.  Basically,  the  plan  specifies  that  questionnaires 
will  be  sent  to  all  initial  course  graduates  and  their 
supervisors  and  that  personal  interviews  will  be  conducted  at 
sites  selected  by  the  Navy.  The  P-3  Instructional  System 
Management  Plan  makes  additional  provisions  to  provide 
questionnaires  to  all  graduates  which  then  makes  the  QCP 
process  basically  the  same  as  the  IPISD  Model. 

The  IPISD  Model  suggests  that  a  random 
sample  of  graduates  and  their  supervisors  be  picked  for 
interviews.  The  QCP,  as  it  presently  exists,  leaves  this 
process  up  to  the  Navy.  To  maintain  the  least  amount  of  bias 
and  ensure  an  accurate  sample  population,  an  expert  should 
probably  be  consulted  to  aid  in  selecting  the  sample. 

Recommendations:  The  QCP  should  be 
structured  to  provide  questionnaires  to  all  course  graduates. 
Additionally,  the  system  should  include  measures  ensuring  a 
high  return  rate  for  the  questionnaires.  Procedures  for 
selection  of  a  sample  population  to  be  interviewed  should  be 
devised  by  experts  and  structured  such  that  Navy  evaluation 
personnel  could  carry  out  the  procedures  on  a  continuous 
basis. 
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b.   Collect  Data 

Once  a  detailed  external  evaluation  plan  has  been 
completed,  procedures  for  gathering  data  become  relatively 
straight  forward.  The  actual  data  collection  falls  into  five 
major  categories:  (1)  baseline  data,  (2)  job  performance 
evaluation  data,  (3)  questionnaire  data,  (4)  personal 
interview  data,  and  (5)  records  of  students1  performance 
while  enrolled  in  the  training  program.  The  procedures  for 
data  collection  in  each  category  are  detailed  below. 

(1)  Collect  Baseline  Date.  The  recommended  col- 
lection of  baseline  data  takes  place  in  Block  II. 2  as  shown 
in  Figure  5.1.  The  data  is  used  both  to  confirm  the  need  for 
the  development  of  new  instruction  and  for  use  in  conducting 
external  evaluation  of  the  new  instruction.  To  fulfill  its 
purpose,  the  baseline  study  must  satisfy  three  basic 
requirements.  First,  the  data  gathering  devices  (JPMs, 
etc.),  must  gather  valid  and  reliable  information.  Second, 
if  JPMS  are  administered,  the  sample  group  must  be 
representative  of  the  population  to  whom  the  results  will 
apply.  Third,  to  ensure  accurate  data  gathering,  the  sample 
group  must  not  be  substantially  different  from  the  group 
involved  in  the  external  evaluation. 

Selecting  an  accurate  sample  population  is 
fairly  complex  and  possibly  should  be  left  to  experts.  The 
IPISD    Model    should   be   consulted   for    additional 
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recommendations  and  guidelines  on  selecting  sample  sizes  and 
sampling  procedures,  etc. 

Discussion:  The  IPISD  Model  relies  on  the 
utilization  of  previously  constructed  JPMs  to  perform  its 
baseline  study.  The  results  of  this  study  may  then  be  used 
as  data  in  the  external  evaluation  process.  Earlier  in  this 
section  it  was  pointed  out  that  due  to  a  difference  in  design 
techniques,  the  development  of  the  P-3  Instructional  System 
did  not  include  the  construction  of  complete  job  performance 
measures.  Therefore,  at  the  present  time,  a  baseline  study 
for  the  P-3  Instructional  System,  based  upon  JPMs  does  not 
and  cannot  exist. 

Recommendations:  Baseline  data  plays  a  key 
role  in  external  evaluation.  The  data  producing  study,  if 
conducted  properly,  provides  a  standard  to  which  the  other 
sources  of  data  may  be  compared.  The  ability  to  make  this 
comparison  provides  a  stable  tool  which  tends  to  improve  the 
worth  of  the  evaluation.  Therefore,  it  is  recommended  that 
steps  be  taken  to  determine  what  data  exists  within  the  P-3 
Instructional  System  that  could  be  used  in  a  baseline  study. 
Measures  should  then  be  taken  to  produce  formal  baseline  data 
suitable  for  use  in  external  evaluation.  The  study  should 
adhere  to  the  sample  population  selection  procedures 
discussed  in  the  IPISD  Model,   and  should  contain   background 
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data  similar  to  that  required  in  the  ANALYZE  JOB  Block  of  the 
IPISD  Model. 

(2)  Collect  Job  Performance  Evaluation  Data. 
Block  1.3  of  the  IPISD  Model  (Figure  5.1)  states  that  there 
must  be  some  kind  of  JPM  developed  for  every  task  selected 
for  training.  Without  these  valid  JPMs,  there  will  be  no 
hard  data  (i.e.,  no  direct  measure)  of  graduates1 
performance.  Non-utilization  of  JPMs  bases  the  external 
evaluation  on  interviews  and  impressionistic  kinds  of  data 
which  make  decisions  far  riskier  than  those  connected  with 
JPMs. 

Soundly  constructed  JPMs  can  often  be  used 
to  overcome  the  constraints  of  time,  equipment,  facility 
availability,  and  cost  encountered  in  actual  observation  of 
graduates  on-the-job,  for  external  evaluation.  When  these 
constraints  make  it  impractical  to  evaluate  the  actual 
performance  of  the  tasks  for  which  the  graduates  were 
trained,  the  evaluation  can  be  based  on  the  JPMs. 
Additionally,  utilizing  JPMs  for  evaluation  allows  them  to  be 
compared  to  the  actual  job  requirements,  thus  allowing  their 
revisions,  when  necessary,  to  ensure  successful  job 
performance. 

Discussion:  The  IPISD  Model  relies  heavily 
upon  JPM  to  provide  a  means  of  directly  measuring  a 
graduate's  on-the-job  performance.    There  is   little  doubt 
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that  well  constructed  JPMs  provide  the  optimum  tool  for  data 
comparison,  baseline  studies,  and  even  performance  evaluation 
when  actual  observation  of  the  graduate  doing  his  job  is  not 
practical.  The  lack  of  adequately  constructed  JPMs  for  each 
task  in  the  P-3  Instructional  System  make  their  use  for 
conducting  evaluation  impossible  at  the  present  time. 
According  to  a  P-3-Instructional  System  Evaluation  Study 
conducted  by  Pacer  Systems,  Inc.  for  the  Naval  Training  and 
Equipment  Center  [Ref.  19],  instructional  system  job 
performance  measures  are  "Inadequately  stated  or  not 
appropriate  to  tasks,  or  non-existent". 

To  overcome  the  problem  of  inadequate  and 
non-existent  JPMs  within  the  P-3  Instructional  System,  one  of 
two  possible  steps  must  be  taken.  First,  solidly  constructed 
JPMs  can  be  provided,  or  second,  the  QCP  can  rely  upon  the 
so-called  "riskier"  methods  of  gathering  data  for  external 
evaluation. 

Recommendations:  An  immediate  study  should 
be  conducted  by  qualified  personnel  to  determine  the  exact 
status  of  JPMs  in  the  P-3  Instructional  System.  The  study 
should  focus  upon  the  cost  and  time  to  provide  adequate  JPMs 
for  each  task  selected  for  training  within  the  system.  The 
results  of  the  study  should  then  be  compared  to  possible  use 
of  alternative  methods  of  data  gathering,  such  as 
questionnaires  and  personal  interviews.  The  comparison  should 
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concentrate  on  such  areas  as  cost,  shortest  time  to 
incorporate,  accuracy  of  data,  and  long-term  effectiveness  of 
the  data-gathering  devices. 

(3)  Collect  Questionnaire  Data.  Although  mailed 
questionnaires  are  not  the  most  reliable  method  of  gathering 
data  from  graduates  and  their  supervisors,  they  are  the  least 
expensive.  Meticulous  attention  to  detail  is  required  in 
questionnaire  construction  and  utilization  to  ensure  that 
sufficient  quantities  of  accurate  data  are  received  for 
evaluation.  When  constructing  the  questionnaires,  a  few 
general  rules  should  be  kept  in  mind:  (1)  ask  specific 
questions,  (2)  list  the  tasks  the  graduate  was  trained  to 
perform  in  the  main  body  of  the  questionnaire,  and  (3)  supply 
open-ended  questions  asking  for  suggestions  on  improving  the 
training  program. 

A  large   sample   is   required   to  provide 
accurate   data.    Therefore,   a  total  population  sampling   is 
desirable   to  help  ensure   a  large  enough   return   rate   to 
overcome  variations  in  job  requirements  and  erratic   behavior 
of  respondents. 

Discussion:  The  QCP  states  that  question- 
naires are  to  be  sent  to  all  course  graduates  and  their  on- 
the-job  supervisors.  Sample  questionnaires  designed  for  this 
purpose  are  contained  in  Appendix  A  for  the  course  graduate, 
and  in  Appendix  C  for  his  supervisor. 
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Recommendations:  Properly  constructed 
questionnaires  utilized  in  a  well-managed  distribution  and 
retrieval  program  can  be  instrumental  in  gathering  accurate 
evaluation  data  in  large  quantities.  Should  the  P-3 
Instructional  System  management  personnel  make  the  decision 
to  use  alternate  data  gathering  methods  instead  of  JPMs  for 
their  primary  data  collection  tools,  the  quality  and 
management  plan  for  questionnaires  will  become  critical. 
Therefore,  it  is  recommended  that  the  presently  existing  QCP 
sample  questionnaires  be  reviewed  for  construction  accuracy 
and  proper  content.  A  questionnaire  should  then  be  devised 
for  each  aircrew  position  covered  by  the  P-3  Instructional 
System. 

(4)  Collect  Personal  Interview  Data.  Special- 
ists from  the  quality  control  team  who  are  familiar  with  how 
the  graduate  was  trained,  are  usually  best  suited  to 
interview  the  graduate  and  his  supervisor.  Separate 
interviews  with  the  selected  sample  populations  of  graduates 
and  their  supervisors  should  be  structured  to  provide  data  on 
graduate  assignments,  utilization,  and  work  proficiency.  If 
necessary,  telephone  interviews  may  be  used. 

The  interviews  should  consist  of  pre-planned 
lists  of  questions,  such  as  those  covered  earlier  in  this 
section,  used  for  the  primary  purpose  of  determining  the 
graduate's  proficiency.    Also  the  interview  should  consider 
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how  the  graduates'  skills  are  being  utilized  and  how  well  the 
graduate  is  handling  additional  training.  The  structure  of 
the  interview  should  be  kept  flexible  enough  to  fit  the 
responses  obtained  from  the  person  being  interviewed. 

Discussion:  Earlier  in  this  section  it  was 
mentioned  that  the  QCP  provides  for  on-site  interviews  with 
course  graduates  and  their  on-the-job  supervisors.  Also  the 
plan  specifies  that  the  sample  to  be  interviewed  should  be 
selected  by  the  Navy.  The  QCP  does  not  provide  any  detailed 
procedures  for  selecting  a  sample  population  or  for  preparing 
and  conducting  interviews. 

The  most  useful  information  is  usually 
obtained  from  an  interview  that  is  planned  and  conducted  by 
an  expert  in  the  field.  The  interviewer  should  not  only  be 
well  versed  in  interviewing  techniques,  but  should  also  be 
familiar  with  the  particular  job  held  by  the  graduate  he  is 
interviewing. 

Once  again  it  should  be  pointed  out  that  if 
P-3  Instructional  System  Management  Personnel  choose  methods 
for  data  gathering  other  than  JPMs,  then  information  from 
interviews  will  play  a  significant  role  in  the  external 
evaluation  process.  Therefore,  construction  of  the  interview 
structure,  and  the  manner  in  which  it  is  presented,  should  be 
accomplished  in  the  most  optimum  fashion  allowable,  within 
imposed  constraints. 
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Recommendations:  Prior  to  the  implementa- 
tion of  the  QCP,  experts  should  be  consulted  to  recommend  a 
basic  interview  structure  for  each  of  the  aircrew  positions 
trained  in  the  P-3  Instructional  System.  Additionally, 
training  should  be  provided  on  interviewing  techniques  to 
those  who  will  travel  to  the  field  to  conduct  the  interviews. 
For  example,  the  evaluation  teams  that  were  proposed  in  the 
introduction  section  of  this  chapter. 

The  selection  of  sample  populations  and 
sampling  procedures,  as  mentioned  earlier,  should  probably  be 
left  to  personnel  knowledgeable  in  that  field.  Otherwise, 
bias  and  inaccurately  gathered  data  would  no  doubt  degrade 
the  value  of  information  gained  from  the  interviews. 

(5)  Obtain  Records  of  Students'  Performance 
During  Instruction.  Records  of  student's  performance  should 
be  available  to  the  external  evaluator  in  case  the  other 
evaluation  data  indicate  the  students  are  not  performing 
satisf actorilly  on  the  job.  This  data  supplies  information 
such  as:  (1)  students'  scores  on  each  lesson  post-test,  (2) 
passing  criteria  for  each  test,  and  (3)  the  number  of  tries 
the  students  required  to  pass  each  test.  The  construction  of 
the  performance  data  sheet  also  allows  students'  scores  to  be 
easily  compared  in  an  effort  to  locate  unfavorable  trends. 

Discussion:  The  P-3  Instructional  System 
Management   Plan   specifies   that   records   of    students' 
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performance  during  training  be  maintained  on  "Student  Test 
Score  Transmittal  Sheets".  Shortly  after  implementation  of 
their  new  ISD  training  program,  VP-31  began  utilizing  a  word 
processing  system  for  data  handling.  This  system  has  now 
replaced  the  "Student  Test  Score  Transmittal  Sheet"  as  a 
means  of  compiling  and  monitoring  student's  performance  while 
under  instruction.  The  word  processor  seems  to  be  very 
efficient  and  accurate  and  should  fit  well  into  an  external 
evaluation  program. 

Recommendation:  The  use  of  a  word  processor 
has  the  capability  of  dramatically  reducing  the 
administrative  workload  for  ISD  personnel.  There  seems  to  be 
no  reason  to  prevent  this  system  from  being  used  as  an 
integral  part  of  an  external  evaluation  plan.  When  the  QCP 
is  being  completed,  the  word  processing  system  should  be 
checked  to  ensure  that  the  proper  data  are  being  gathered, 
and  that  the  data  are  being  provided  in  a  format  that  will  be 
usable  in  the  evaluation  plan. 

c.   Consolidate  Data  and  Make  Recommendations 

The  consolidation  of  data  into  a  usable  format  is 
covered  in  Block  I . 1  of  the  IPISD  Model.  Using  this  format, 
graduate  questionnaire  data,  supervisor  questionnaire  data, 
graduate  interview  data,  supervisor  interview  data  and  JPMs 
may  be  compiled  separately.   Evaluation  can  then  be  initiated 
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by  asking  some  pertinent  questions,  and  checking  the  blocks 
of  data  for  answers. 

Normally  the  first  question  to  ask  should  be,  can 
the  job-holding  graduate  perform  the  tasks  he  was  trained  to 
perform  at  the  planned  level  of  proficiency?  This  question 
usually  leads  to  other  questions  such  as:  exactly  which 
tasks  are  not  being  properly  performed,  and  what  does  the  job 
performance  evaluation  say  about  these  unsatisf actorilly 
performed  tasks? 

A  few  additional  questions  to  ask  are: 

(1)  Are  the  tasks  identified  in  the  original  job  ana- 
lysis the  same  as  those  tasks  the  supervisors  think  are 
necessary  to  the  job? 

(2)  Are  there  areas  where  the  graduates  can  score  well 
on  the  JPMs  but  can't  do  the  actual  task? 

The   above  questions  are  provided   to   illustrate 

that   there   are   no  specific  rules  for  evaluating   data   and 

making  recommendations  for  change.    Above  all,   common  sense 

should  be  used  when  evaluating  the  data.    If  most  graduates 

and   supervisors   are   satisfied  with   the  quality  of   the 

training   program,   very  few,   if   any  changes   are   usually 

necessary.   On  the  other  hand,  if  a  significant  number  of  the 

sample  population  feels  that  training  quality  is  low  in   some 

area,   review  the  data   to  find  possible  causes   for   the 

inadequacy.    The  IPISD  Model  should  be  consulted  for  several 

more  general  suggestions  for  evaluating  data  and  correcting 

problems. 
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As  a  result  of  the  above  evaluation  techniques, 
changes  may  be  made  to  the  instructional  system  and/or  the 
job  structure.  As  an  example  of  how  the  process  works,  if 
external  evaluation  shows  problems  with  the  instructional 
program,  one  would  then  go  to  Block  V.3  REVISE  SYSTEM,  of  the 
IPISD  Model,  and  make  the  necessary  changes.  Next  internal 
and  external  evaluation  would  be  conducted  again  to  check  for 
errors.  Eventually  the  only  program  revisions  that  will  be 
necessary  are  those  resulting  from  changes  in  content  and 
doctrine. 

Once  this  point  is  reached,  the  effectiveness  of 
the  training  program  should  be  checked  by  once  again  seeking 
answers  to  the  questions: 

(1)  Do  graduates  of  the  current  training  program  show 
better  performance  on  the  job  than  graduates  of  past 
programs? 

(2)  Do  graduates  of  the  current  training  program  re- 
quire less  on-the-job  training  than  graduates  of  other 
courses? 

To  answer  the  first  question,   a  sample  from   the 

baseline  study  that  is  as  similar  as  possible  to  graduates  of 

the  instruction  being  evaluated  is  selected.    The  two  groups 

are   then  compared  on  their  performance  on  the  JPMs.   If   the 

performance   level  of  the  two  groups  is  close  (say  75  verses 

85  percent)   experts   should   construct   and   administer   a 

statistical   test   to  determine  how  often  such   a  difference 

could   be   due   to  chance  alone.    The  answer  to   the   second 
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question  can  be  obtained  in  a  similar  manner,  but  the  data  is 
gained  from  the  questionnaires  and  interviews. 

Discussion:  Following  the  data  gathering 
effort,  of  external  evaluation,  are  the  extremely  important 
steps  of  data  consolidation  and  analysis.  Together,  data 
gathering  and  data  analysis  provide  the  basic  framework  for 
the  entire  concept  of  conducting  external  evaluation. 
Therefore,  it  is  only  proper  that  optimum  analysis  of  the 
data  be  provided  after  extensive  efforts  have  been  expended 
to  gather  the  "correct"  data. 

Although  the  QCP  does  not  specify  adequate 
procedures  for  data  consolidation  and  analysis,  at  the 
present  time,  it  does  provide  some  general  guidelines.  The 
plan  places  the  responsibility  for  data  analysis  upon  the 
Instructional  System  (IS)  Department  of  VP-31.  The  basic 
process  of  data  summarization  and  analysis  within  this 
department  falls  on  the  Subject  iMatter  Experts  (SMEs)  ,  for 
each  aircrew  position.  The  basics  for  the  SMEs  analysis  is  a 
comparison  of  fleet  input  data  to  other  data  (i.e.,  data  from 
present  students  and  instructors) .  The  purpose  of  the 
analysis  is  to  determine  the  performance  of  a  graduate  on- 
the-job,  and  to  point  out  areas  within  the  training  program 
where  the  subject  matter  content  is  deficient.  Once  the  SME 
has  completed  his  analysis,  he  suggests  corrective  actions 
that   will   be  considered  by   Training   Officers/Course 
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Supervisors.  Many  of  the  problems  which  arise  during 
analysis  and  revision  attempts  will  be  handled  by  a 
Curriculum  Control  Board  (CCB) ,  which  is  composed  of 
representatives  from  the  IS  Department,  FASO,  VP-31  training 
office,  etc.  These  procedures  are  not  detailed  enough  to 
conduct  an  effective  analysis,  but  they  do  seem  to  follow  the 
basic  recommendations  of  the  IPISD  Model. 

Recommendations:  Regardless  of  whether  external 
evaluation  of  the  P-3  Instructional  System  is  conducted  by  an 
evaluation  team,  as  suggested  by  the  author,  or  by  SMEs  from 
the  IS  Department,  detailed  administrative  and  analysis 
procedures  must  be  provided  if  the  evaluation  is  to  be 
successful.  The  procedures  must  be  explicit  enough  to 
overcome  the  high  job  turnover  rate  of  professional 
evaluation  team  members  and  SMEs,  and  must  provide  additional 
guidelines  to  compensate  for  non-professional  evaluators 
(i.e.,  SMEs  who  have  not  been  extensively  trained  in 
interviewing,  data  interpretation,  etc.)  if  they  are  used 
instead  of  an  evaluation  team. 

The  process  for  completing  the  QCP  should  provide 
detailed  procedures  for  questionnaire  distribution,  retrieval 
of  all  data,  consolidation  and  analysis  of  data,  and  use  of 
data  for  making  instructional  system  revisions.  The 
procedures  must  also  take  into  account  whether  or  not  the  use 
of   JPMs  have  been  employed  in  the   evaluation.   Several   of 
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these   areas   are  addressed  briefly  in  the  IPISD  Model,   but 
much  more  detail  must  be  provided  if  a  realistic  and  accurate 
evaluation  is  expected  to  take  place  on  a  continuous  basis. 
3.   Outputs 

a.  Products 

The  primary  product  of  the  evaluation  process  is 
an  external  evaluation  report  (EXER) .  This  report  is  composed 
of  a  summary  statement  of  all  external  evaluation  procedures, 
findings,  interpretations,  and  recommendations  for  revisions. 

b.  Other  Documentation 

This  documentation  includes  supporting  informa- 
tion such  as: 

(1)  A  statement  giving  pertinent  information  about 
the  graduate  being  evaluated. 

(2)  The  date  evaluation  took  place,  and  the  length  of 
time  between  the  completion  of  training  and  the  conduction 
of  external  evaluation. 

(3)  A  statement  describing  how  the  evaluation  was  con- 
ducted. 

(4)  A  statement  describing  graduates'   responses  to 
questionnaires  and  interviews. 

(5)  Job  performance  evaluation  results. 

(6)  How  the  evaluation  team  interpreted  the  data   from 
items  (3),  (4)  and  (5)  above. 

(7)  Other  information  that  influenced  final   recommen- 
dations. 

(8)  Recommendation  for    revising   the    insructional 
system. 
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(9)  Recommendations  for  revising  the  job  structure. 

(10)  General  rcommendations  not  covered  above. 

Discussion:  Since  the  QCP  has  not  been 
completed,  it  does  not  address  the  external  evaluation  report 
and  its  supporting  information.  However,  the  plan  does 
recognize  the  need  for  such  documentation,  in  that  it 
discusses  such  products  as  a  Revision  Specification  Worksheet 
and  the  necessity  for  course  revisions. 

Recommendations:  The  IPISD  Model  seems  to 
provide  a  complete  list  of  external  evaluation  outputs.  It 
is  recommended  that  QCP  designers  refer  to  the  IPISD  Model 
for  suggestions  in  completing  this  phase  of  the  external 
evaluation  plan. 

B.   FEASIBILITY  OF  INCORPORATING  THE  P-3  INSTRUCTIONAL  SYSTEM 
EXTERNAL  QUALITY  CONTROL  PLAN 

The  ISD  based  P-3  Instructional  System  was  implemented  at 
VP-31  approximately  three  and  one-half  years  ago.  Since  that 
time  IS  Department  management  and  instructional  personnel 
have  succeeded  in  correcting  almost  all  of  the  design  errors 
and  implementation  problems  associated  with  the  new  program. 
These  corrections  along  with  a  series  of  continuous  material 
and  procedural  improvements  have  allowed  the  system  to  evolve 
into  a  state  of  extreme  stability. 
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The  stable  condition  of  the  P-3  Instructional  System  plus 
the  improved  abilities  of  its  managers  and  instructors,  since 
implementation,  suggest  the  present  time  as  being  favorable 
for  incorporation  of  the  evaluation  phase  of  the  program. 
This  important  step  would  provide  the  VP  FRSs  the  "closed- 
loop"  training  system  they  initially  sought  to  replace  the 
traditional  system.  Since  this  thesis  focuses  on  external 
evaluation,  this  discussion  will  be  limited  to  that  part  of 
instructional  system  design  and  operation. 

Although  this  appears  to  be  an  opportune  time  to  complete 
and  incorporate  the  QCP,  several  obstacles  exist  that  must  be 
overcome  or  at  least  circumvented.  One  of  the  obstacles 
hindering  successful  implementation  and  operation  of  the  QCP 
is  the  lack  of  enough  adequately  trained  personnel  to  act  as 
evaluators/administrators.  No  matter  which  evaluation 
concept  is  adopted  (i.e.,  using  an  evaluation  team  or 
existing  IS  Department  personnel  for  evaluators) ,  there  are 
not  enough  suitable  personnel  presently  available  to 
adequately  do  the  job.  This  problem  seems  to  have  the 
highest  probability  of  being  solved  by  adopting  the  team 
concept.  For  example,  an  instructional  design/evaluation 
expert  could  be  hired  on  a  permanent  basis  to  provide  team 
stability.  The  team  members  could  then  be  obtained  in  the 
following  manner.  An  individual  is  placed  within  the  FRS  as 
an  SME/track  manager   for  approximately  one   and  one-half 
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years.  Following  this  time  period,  he  is  then  rotated  to  the 
external  evaluation  team  for  one  and  one-half  years.  This 
method  would  provide  a  knowledgeable  individual,  for  every 
aircrew  position,  to  act  as  an  external  evaluator .  The 
instructional  design/evaluation  expert  and  turn-over 
notebooks  would  be  available  to  bring  the  newly  installed 
evaluators  "up- to- speed"  on  all  procedures  that  deal  with 
evaluation  such  as  interviewing  techniques,  data  analysis, 
etc.  This  suggestion  for  forming  an  evaluation  team  is  based 
on  the  premise  that  experts  have  been  hired  to  totally 
complete  and  aid  in  the  initial  implementation  of  the  QCP. 
This  concept  could  be  duplicated  if  necessary  to  provide  a 
complete  team  for  each  FRS.  Depending  upon  the  actual 
workload  involved,  the  teams  could  possibly  assist  in 
conducting  internal  evaluation  for  the  training  system.  This 
suggestion  for  an  independent  evaluation  team  could  possibly 
eliminate  the  shortage  of  suitably  trained  personnel  to 
conduct  external  evaluation. 

A  second  obstacle  that  must  be  faced  in  establishing  a 
fully  functional  QCP  is  inadequate  funding.  With  today's 
"hard-times'*  economic  situation,  getting  funding  for  anything 
short  of  an  emergency  situation  is  next  to  impossible.  Yet, 
one  must  consider  the  benefits  to  be  gained  from  a  fully- 
implemented  state-of-the-art  training  system  (i.e., 
continuous   assurance  that  the  training  program  is  optimally 


91 


effective  and  cost  efficient) .  The  actual  cost  for 
completion  and  implementation  of  the  QCP  appears  to  be  very 
small  when  compared  to  the  resultant  benefits  of  such  a  plan. 
Perhaps  consideration  should  be  given  to  the  IPISD  Model 
statement  [Ref.  8]  that  implementation  is  an  evolutionary 
process.  It  might  be  feasible  to  start  out  conducting 
partial  external  evaluation  and  work  up  to  a  full-fledged 
program  after  some  period  of  time.  According  to  the  IPISD 
Model  [Ref.  8],  "...the  spirit  of  the  procedure  is  far  more 
important  than  the  letter". 

A  third  obstacle  that  must  be  overcome  is  an  inadequate 
management  plan.  The  existing  management  plan  was  designed 
for  use  without  a  complete  QCP  incorporated.  The  IPISD  Model 
stresses  the  necessity  of  matching  the  management  plan  and 
the  delivery  system  to  lower  internal  costs.  This  applies  to 
the  evaluation  process  as  well  as  the  rest  of  the  system. 
Therefore,  the  P-3  Instructional  System  Management  Plan  must 
be  revised  when  a  complete  QCP  is  incorporated. 

The  fourth  and  final  area  that  will  be  addressed 
concerning  incorporation  of  the  QCP  is  the  need  for  improved 
data  handling  capabilities.  Once  the  FRSs  begin  conducting 
external  evaluation,  the  IS  Department  must  have  a  means  to 
store  and  utilize  the  additionally  generated  data.  The 
solution  to  this  problem  could  possibly  be  the  Aviation 
Training   Support  System  (ATSS)  which  is  already  partially  in 
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existence  at  VP-31's  IS  Department.  Presently,  the  ATSS  is 
incomplete  and  the  existing  software  is  very  weak.  However/ 
it  appears  that  acquiring  the  components  to  complete  the 
system,  and  designing  necessary  software  to  handle  the  new 
demands  imposed  by  the  incorporation  of  the  QCP  could  be  both 
possible  and  practical. 
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VI.   DISCUSSION  AND  SUMMARY  OF  RECOMMENDATIONS 

The  P-3  Instructional  System  appears  to  be  a  well- 
constructed  and  expertly  managed  training  program  that 
produces  a  quality  graduate  who  is  thoroughly  capable  of 
doing  his  job  in  the  fleet.  The  problem  that  exists  is  to 
prove  this  statement.  Without  a  viable  external  evaluation 
plan,  it  is  almost  impossible  to  determine  if  the  P-3 
Instructional  System  is  superior  to  the  traditional  training 
program  that  it  replaced  and  if  the  system  is  in  fact 
producing  the  high  caliber  graduate  needed  to  operate  and 
maintain  the  sophisticated  equipment  on  the  P-3  aircraft. 

The  intentions  of  this  thesis  have  been  to  examine  the 
status  of  the  P-3  Instructional  System  External  Quality 
Control  Plan  and  to  provide  recommendations  for  its 
completion  and  incorporation.  The  purpose  for  the 
examination  of  the  Plan  was  to  aid  in  the  improvement  of  the 
instructional  system  and  the  quality  of  its  graduates.  The 
author  realizes,  as  stated  earlier,  that  the  external 
evaluation  plan  cannot  actually  be  thought  of  as  a  single 
unit  of  the  instructional  system,  but  must  be  viewed  as  an 
integral  part  of  the  entire  system.  This  idea  was  kept  in 
mind  during  the  analysis  of  the  plan. 
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During  the  interim  of  this  report,  it  was  pointed  out 
that  due  to  certain  unavoidable  constraints,  the  external 
quality  control  plan  was  never  completed.  Although  this  fact 
is  true,  it  is  the  author's  opinion  that  Courseware,  Inc.  has 
provided  a  reasonably  good  start  on  the  framework  of  the  plan 
when  compared  to  the  IPISD  Model.  Additionally,  the  author 
believes  that,  due  to  the  extreme  importance  of  external 
evaluation  and  the  long  period  of  time  that  the  Instructional 
System  has  been  allowed  to  operate  without  it,  experts  should 
be  employed  immediately  to  complete  the  plan  and  aid  in  its 
incorporation.  There  is  little  doubt  that  this  plan  would 
more  than  pay  for  itself  in  the  long  run. 

During  the  analysis  of  the  QCP,  it  was  noted  that  neither 
ISD  model  provided  detailed  procedures  for  conducting 
external  evaluation.  The  IPISD  Model  stated  that  this  lack 
of  detail  was  intentional  and  would  allow  each  command  to  fit 
the  model  to  its  own  particular  instructional  situation. 
This  approach  is  probably  a  good  idea,  but  the  author 
strongly  believes  that  the  details  for  evaluation  should  be 
provided  by  experts  and  not  Naval  personnel  who  are 
unfamiliar  with  instructional  design  and  evaluation.  It  is 
recommended  that  experts  provide  all  procedural  detail  (i.e., 
general  evaluation  concepts,  data  collection,  interviewing 
procedures,  data  analysis,  etc.)  and  aid  in  the  incorporation 
of   the  plan  before  turning  the  program  over  to  the  VP-31   IS 
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Department.  This  will  provide  a  solid  foundation  for  the 
evaluation  and  should  minimize  errors  in  design  concepts  and 
operating  procedures.  It  should  also  be  experts  that  decide 
whether  it  is  feasible  to  construct  and  use  JPMs  for 
evaluation  data,  or  whether  the  evaluation  should  be  based  on 
other  data,  such  as  questionnaires  and  interviews. 

Instructional  Systems  Development  External  Evaluation  is 
a  precise  and  time-consuming  endeavor.  Since  it  is  a 
possibility  that  completion  and  incorporation  of  the  QCP  may 
not  be  feasible  at  the  present  time,  two  alternative 
evaluation  methods  will  be  mentioned  for  consideration 
First,  Scriven  [Ref.  20]  proposed  evaluation  procedures  whose 
purpose  is  based  on  "goal-free  evaluation".  In  other  words, 
the  evaluation  examines  the  effects  of  an  educational 
innovation  and  assesses  the  worth  of  the  effects,  whatever 
they  are.  With  this  method,  the  evaluator  is  not  confined  to 
the  stated  objectives,  but  is  free  to  assess  and  evaluate 
outcomes  of  any  sort.  This  method  is  complex,  but  allows 
systematic  judgment  of  appropriateness  and  general  worth  of 
the  instructional  product.  The  major  emphasis  of  the  model 
is  verified  performance. 

Second,  Stuff lebeam  [Ref.  21]  developed  evaluation 
methods  composed  of  a  model  called  CIPP  (Context,  Input, 
Process,   and  Product) .    The  model  considers  evaluation  as  a 
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continuous  process  whose  purpose  is  to  guide  decision  making. 
The  major  emphasis  of  the  model  is  on  continuous  planning. 

Although  the  two  evaluation  models  just  discussed  are 
widely  accepted  in  the  field  of  instructional  program  design, 
the  author  feels  that,  if  at  all  possible,  the  P-3 
Instructional  System  should  contain  an  incorporated 
evaluation  plan  that  forms  a  closed-loop  system.  Using 
anything  less  than  pure  ISD  techniques  for  evaluation  would 
probably  be  a  step  backwards  for  the  training  system. 

The  following  recommendations  for  completing  and 
incorporating  the  QCP  are  considered  to  be  important: 

—  A  team  capable  of  providing  continuous  external  eval- 
uation of  the  P-3  Instructional  System  should  be  formed. 
Optimally  the  team  should  not  be  connected  to  the  training 
program. 

A  study  should  be  conducted  to  determine  whether  the 
primary  data  source  for  the  external  evaluation  should  be 
based  on  job  performance  measures  or  alternate  methods  such 
as  questionnaires  and  interviews. 

The  QCP  should  require  external  evaluation  to  be  con- 
ducted 30  to  90  days  after  a  graduate  reports  to  his  job,  as 
recommended  by  the  IPISD  Model. 

—  A  study  should  be  conducted  to  develop  baseline  data 
to  be  used  as  a  standard  for  comparison  in  the  external 
evaluation. 
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—  The  currently  existing  QCP  sample  questionnaire  for 
graduates  and  their  on-the-job  supervisors  should  be  reviewed 
for  accuracy  of  construction  and  content.  A  questionnaire, 
for  each  aircrew  position  trained,  should  be  developed  based 
on  the  updated  sample. 

—  Detailed  interviewing  procedures  and  correct  and  ac- 
curate methods  for  selecting  sample  populations  should  be 
provided  by  experts. 

Generally  speaking,  procedures  for  the  entire  evalu- 
ation process  consisting  of  planning,  data  gathering,  data 
analysis,  and  recommending  revisions  should  be  provided  by 
experts.  This  provision  will  give  the  evaluation  effort  a 
solid  foundation  and  prevent  evaluation  errors. 

—  The  P-3  Instructional  System  Management  Plan  should 
be  updated  to  deliver  the  new  QCP. 

—  A  study  should  be  conducted  to  determine  if  the 
Aviation  Training  Support  Systems  (ATSS)  is  capable  of 
handling  the  data  generated  by  conducting  external 
evaluation. 

The  ISD  based  P-3  Instructional  System  represents  a  major 
step  forward  toward  the  application  of  "state-of-the-art" 
instructional  design  concepts.  Continued  refinement  and  an 
improved  understanding  of  the  ISD  process  should  improve  the 
instructional  system  by  ensuring  optimal  training 
effectiveness,  time  efficiency  and  cost. 
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APPENDIX  A 

INITIAL  COURSE  GRADUATE  FOLLOW-UP  QUESTIONNAIRE 
(Ref.  17:   p.  159) 


You  are  being  asked  to  participate  in  the  evaluation  of 
the  P-3  ISD  training  program.  Please  contribute  your  honest 
views  on  the  training  you  received  prior  to  your  current 
assignment.  All  feedback  will  be  considered  in  specifying 
revisions  to  the  instructional  materials.  Your  help  is 
necessary  for  improving  the  training  program  for  future 
students. 

Please  complete  the  following  information.  All  question- 
naire responses  will  be  considered  confidential. 

Name  

Rank 


Current  Job  Title 
Supervisor's  Name 


Graduate  of  P-3  ISD  Course  Location 
Class  No. 


1.   How   long   have   you  been  working  at  your   present   job? 
(Circle  one) 


a. 

1 

month 

b. 

2 

months 

c. 

3- 

-4  months 

d. 

5- 

-6  months 

2.   How  well  do  you  feel  you  are  able  to  perform  your  present 
job? 

a.  Above  average 

b.  Average 

c.  Below  average 
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Have  you   received  any  on-the-job   training   for   your 
present  job  since  you  arrived? 

a.  Yes 

b.  No 

If  yes,  please  specify: 
How  much? 


What  kind? 


4.  How  well  did  the  P-3  ISD  training  you   received  prepare 
you  for  your  present  job?   (Circle  one) 

a.  More  than  adequate 

b.  Adequate 

c.  Inadequate 

d.  Poor  preparation 

5.  Were  portions   of   the  instruction   irrelevant   to  your 
present  job? 

a.  Yes 

b.  No 

Please  specify:  


6.   In  your   present  job,   how  often  do  you  use   the   skills 
taught  in  P-3  ISD? 

a.  Frequently 

b.  Occasionally 

c.  Not  at  all 
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Please  specify  if  "not  at  all": 


7.   In  your  present  job,   what  tasks  have  given  you  the  most 
difficulty? 


Please  specify: 


8.  What  parts  of  the  instruction  do  you  think  could  be 
improved  or  added  to  better  prepare  future  students  for 
the  job? 


Please  specify: 


9.   Other  comments: 
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APPENDIX  B 

INSTRUCTIONAL  SYSTEMS  SURVEY 
PILOT  STUDENT 
(Ref.  18:   p.  H-26) 


Within  the  last  six  months  you  completed  P-3  FRS 
training.  You  should  have  now  had  an  opportunity  to  apply 
this  training  as  a  member  of  a  flight  crew.  We  would  like 
you  to  think  back  to  the  training  you  received  at  the  FRS  and 
complete  the  attached  questionnaire.  The  results  of  these 
questionnaires  will  be  used  to  improve  the  training  program 
for  future  students.  Thank  you  for  your  cooperation  in 
completing  and  returning  this  questionnaire  within  three 
weeks  in  the  envelope  provided.   Your  answers  are  IMPORTANT. 
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INITIAL  COURSE  GRADUATE  FOLLOW-UP  QUESTIONNAIRE 


Name 


Rank/Rate 


Aircrew  Position/Designation 
Supervisor's  Name  


Graduate  of  P-3  FRS  Course  Location 
Class  No. 


You  are  being  asked  to  participate  in  the  evaluation  of 
the  P-3  FRS  training  program.  Please  contribute  your  honest 
views  on  the  FRS  training  you  received  prior  to  your  current 
assignment.  Your  help  is  necessary  for  improving  the 
training  program  for  future  students. 

Please  complete  the  following  information.  All 
questionnaire  responses  will  be  considered  confidential. 

1.  How  long  have  you  been  assigned  to  your   present  crew 
position? 

a.  1  month 

b.  2  months 

c.  3-4  months 

d.  5-6  months 

2.  How  well  did  the  P-3  FRS  training  you  received  prepare 
you  for  your  present  aircrew  assignment? 

a.  More  than  adequate 

b.  Adequate 

c.  Inadequate 

If  you  answered  c,  please  specify  which  areas:   
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3.   Do  you  use  the  majority  of  the  skills  learned  in  the   P-3 
FRS  course  on  your  present  aircrew  position? 

a.  Frequently 

b.  Occasionally 

c.  Not  at  all 

Please  specify  those  you  do  not  use:   


4.   In  your  present  aircrew  position,   what  tasks  have  given 
you  the  most  difficulty?   (Please  specify) :   


5.  After  having  been  on-the-job  for  a  few  months,  which  of 
the  following  subject  areas  covered  in  P-3  FRS  training 
would  you  like  to  see  improved?   (   ) 

a.  Systems 

b.  Normal  Procedures 

c.  Performance 

d.  Emergency  Procedures 

e.  Instrument  Procedures 

f .  Safety/Survival/Landing  Procedures 

g.  Navigation 

h.  Oceanography 

i.  Search  Tactics 

j.   Localization  Tactics 

k.         Introduction  to  SS-3  (RADAR,  ESM,  IRDS,  MAD) 

1.   Active  Tactics 

m.   Search/Kill  Stores 

n.   Plot  Stabilization 

o.   Communications 

p.   Secondary   Missions    (Mining,   SAR,   Rigging, 

SOSUS,  Soviet  Naval  Threat,  RECCO,   Coordinated 

Operations) 

Please  specify  the  improvements  you  recommend:   
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6.   Which  of  the  following  performance  situations  in  the   P-3 
FRS  training  would  you  like  to  see  improved?   (   ) 

Device  Sessions 

a.   Cockpit  Procedural   Trainers   (CPT-Normal   and 

Emergency  Procedures) 

b.   Flight  Simulators   (2F87-Normal   and  Emergency 

Procedures) 

c.   Aircraft  Labs  (Preflight/Postf light) 

Flights 

a.   Fly  Phase  (I-V) 

b.   NAV  Phase  (I  and  NAV  Extended) 

c.   TAC  Phase  (I-IV) 

Please  specify  the  improvements  you  recommend:   


7.   Other  comments: 
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APPENDIX  C 

JOB  SUPERVISOR  FOLLOW-UP  QUESTIONNAIRE 
(Ref.  17:   p.  163) 

You  are  being  asked  to  participate  in  the  evaluation  of 
the  P-3  ISD  training  program.  Please  contribute  to  this 
effort  by  evaluating  the  job  performance  of  P-3  ISD  initial 
course  graduates.  All  feedback  will  be  considered  in 
specifying  revisions  to  the  instructional  materials.  Your 
help  is  necessary  to  improve  the  training  for  future  course 
graduates. 

Please  complete  the  following  information.  All 
questionnaire  responses  will  be  considered  confidential. 

Name  

Rank  

Job  Title 


P-3  Course  Graduates  Supervised: 
Names: 


1.  How  long  have  you  supervised  the  P-3  ISD  initial   course 
graduate (s) ?   (Circle  one) 

a.  1  month 

b.  2  months 

c.  3-4  months 

d.  5-6  months 

2.  How  well  are  the  graduates  able  to  perform  on  the  job? 

a.  Above  average 

b.  Average 

c.  Below  average 
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Did   the  graduate (s)  receive  any  additional  on-the-job 
training  after  arrival  to  the  squadron?   (Circle  one) 

a.  Yes 

b.  No 

If  yes,  please  specify: 
How  much? 


What  kind? 

4.   How  do  these  graduates  perform  on  the  job  as  compared   to 
those  who  were  trained  in  the  past? 

a.  Above  average 

b.  Average 

c.  Below  average 

Please  specify  if  below  average:   


5.   In  what   task   areas  were   the   graduates   inadequately 
prepared  to  perform  on  the  job? 


Please  specify: 


Has   the  graduate  had  accidents  or  been  reprimanded   for 
misuse  or  improper  operation  of  equipment? 

a.  Accident (s) 

b.  Misuse  of  equipment 

c.  Improper  operation  of  equipment 
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Please  specify: 


7.   Has   the   graduate   been  warned   for   unusually   bad 
performance? 

a.  Yes 

b.  No 

Please  specify:   


8.  Has   the  graduate   been  commended  or   recommended   for 
promotion? 

a.  Yes 

b.  No 

c.  Possible  future  consideration 

9.  Other  comments:  
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APPENDIX  D 

INSTRUCTIONAL  SYSTEMS  SURVEY 
PILOT  SUPERVISOR 
(Ref.  18:   p.  H-20) 

You  are  currently  supervising  the  recent  FRS  graduate 
whose  name  appears  on  the  enclosed  questionnaire.  You  should 
have  had  some  opportunity  to  observe  his  performance  as  a 
flight  crew  member.  We  would  like  you  to  evaluate  the 
graduate's  performance  by  completing  the  questionnaire.  The 
results  of  these  questionnaires  will  be  used  to  improve  the 
training  program.  Thank  you  for  your  cooperation  in 
completing  and  returning  this  questionnaire  within  three 
weeks  in  the  envelope  provided.   Your  answers  are  IMPORTANT. 
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JOB  SUPERVISOR  FOLLOW-UP  QUESTIONNAIRE 


Name 


Rank/Rate 


Crew  Position/Designation 

P-3  FRS  Course  Name:  

FRS  Graduate  Name: 


You  are  being  asked  to  participate  in  the  evaluation  of 
the  P-3  FRS  training  program.  Please  contribute  to  this 
effort  by  evaluating  the  aircrew  performance  of  P-3  FRS 
course  graduates.  Your  help  is  necessary  for  improving  the 
training  program. 

Please  complete  the  following  information.  All 
questionnaire  responses  will  be  considered  confidential. 

1.  How  long  have  you  supervised  the  FRS  graduate  names 
above?   (Circle  one) 

a.  1  month 

b.  2  months 

c.  3-4  months 

d.  5-6  months 

2.  How  well  is  the  graduate  able  to  perform  his  aircrew 
duties?  (Please  consider  the  following  criteria  in 
answering  this  question) :   (Circle  one) 

-  Qualified    =  Tasks   performed   in   accordance   with 

NATOPS  or  other  established  standards. 

-  Minor  Errors  =  Tasks  NOT  performed  in  accordance   with 

NATOPS  or  other  established  standards 
when  the  performance  does  not  meet  the 
criteria  specified  for  major  errors. 

-  Major  Errors  -  Tasks  performed  incorrectly  AND  which 

have  the  potential  of: 
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1)  Adversely  affecting  safety  of   flight 
or  personnel. 

2)  Risking  damage  to  equipment. 

3)  Jeopardizing  mission  accomplishment. 


a.  Qualified 

b.  Minor  errors 

c.  Major  errors 

Please  specify: 


3.   Did   the  graduate   require  any  additional   training   for 
skills  he  should  already  have?   (Circle  one) 


a.  Yes 

b.  No 

If  yes,  please  specify: 


Is   the   graduate  making  satisfactory  progress   towards 
designation  as  PP3P/PP2P/PPC? 


a.  Yes 

b.  No 

Please  specify: 


5.  Has  the  graduate  had  any  accidents  or  incidents  that  were 
the  result  of  personal  error  or  been  reprimanded  for 
misuse  or  improper  operation  of  equipment? 


a.  Accident(s)  or  incident(s) 

b.  Misuse  of  equipment 

c.  Improper  operation  of  equipment 


111 


Please  specify: 


6.   Has   the   graduate  been  warned  for   unusually  deficient 
performance? 


a.  Yes 

b.  No 

Please  specify: 


7.   Has  the  graduate  exhibited  superior  performance? 

a.  Yes 

b.  No 

Please  specify:  


8.  After  having  supervised  the  graduate  "on-the-job"  for  a 
few  months,  in  which  of  the  following  areas  do  you  feel 
that  this  person  requires  more  training?   (   ) 

Ground  School 

a.  Systems 

b.  Normal  Procedures 

c.  Performance 

d.  Emergency  Procedures 

e.  Instrument  Procedures 

f .  Safety  Survival/Landing  Procedures 

g.  Navigation 

h.  Oceanography 

i.  Search  Tactics 

j.  Localization  Tactics 

k.         Introduction  to  SS-3  (RADAR,  ESM,  IRDS,  MAD) 

1.  Active  Tactics 

m.  Search/Kill  Stores 

n.        Plot  Stabilization 
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o.   Communications 

p.   Secondary   Missions    (Mining,   SAR,   Rigging, 

SOSUS,  Soviet  Naval  Threat,   RCCO,   Coordinated 

Operations) 

Device  Sessions 

a.   Cockpit  Procedural  Trainers   (CPT-Normal  and 

Emergency  Procedures) 

b.   Flight   Simulators   (2F87-Normal  and  Emergency 

Procedures) 

c.   Aircraft  Labs  (Pref light/Postf light) 


Flights 


a.   Fly  Phase  (I-V) 

b.   NAV  Phase  (I  and  NAV  Extended) 

c.   TAC  Phase  (I-IV) 

Please  specify  the  improvements  you  recommend: 


9.   Other  comments: 
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APPENDIX  E 

REVISION  SPECIFICATION  WORKSHEET 
(Ref.  17:   p.  75) 
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